AI Coding Standards for Data Scientists

Why Data Scientists Need AI Coding Rules

You are a data scientist. You write Python in Jupyter notebooks, build data pipelines, and train models. Your code: starts as exploration (quick experiments, prototype transforms) and eventually becomes production (scheduled pipelines, deployed models). The problem: exploration code and production code have different quality requirements — but the transition between them is where bugs, tech debt, and convention violations accumulate. Without AI rules: your notebook code uses pandas with chained operations, hardcoded file paths, and print statements for logging. The production requirement: structured logging, configuration-driven paths, and error handling. The rewrite: 60-80% of the code changes when transitioning from notebook to production.

With AI rules: the AI generates your exploration code using production-ready patterns from the start. AI rule: 'Use logging instead of print. Use pathlib instead of string paths. Use configuration objects instead of hardcoded values.' Your notebook code: already follows production conventions. The transition to production: changes 10-20% of the code (business logic refinements) instead of 60-80% (convention rewrites). The time saved: 4-8 hours per pipeline promotion.

The data-science-specific benefit: AI rules bridge the research-production gap that every data science team struggles with. Research code that already follows production conventions: faster to review, faster to deploy, and fewer production incidents caused by convention mismatches between notebook prototypes and production pipelines.

How AI Rules Standardize Data Pipeline Code

Data pipeline consistency: Data Scientist A reads CSV with pandas, transforms with method chaining, and writes to Parquet. Data Scientist B reads CSV with polars, transforms with SQL-like expressions, and writes to Delta Lake. Both pipelines work. Both are unmaintainable by the other person. AI rule: 'Data pipelines use polars for transforms, read/write Parquet format, and follow the extract-transform-load function pattern: extract_<source>(), transform_<entity>(), load_<destination>().' With this rule: every data scientist's AI generates pipelines with identical structure, library choices, and naming conventions. Any team member can maintain any pipeline.

Schema validation: production data pipelines fail silently when input schemas change. AI rule: 'All pipeline inputs must be validated with pandera or pydantic before processing. Define expected schema as a dataclass. Fail loudly on schema mismatch.' The AI: adds schema validation to every generated pipeline. The data scientist: never deploys a pipeline that silently processes malformed data. The production reliability: guaranteed by the rules, not dependent on each data scientist remembering to add validation.

Experiment tracking: data scientists run experiments with different hyperparameters, datasets, and preprocessing steps. Without conventions: experiments are tracked in spreadsheets, notebooks, or not at all. AI rule: 'All experiments use MLflow for tracking. Log parameters, metrics, and artifacts in every training run. Use the naming convention: experiment_<model>_<dataset>_<date>.' The AI: adds MLflow tracking to every generated training script. The data scientist: has reproducible experiments by default. AI rule: 'Data pipeline rules solve the #1 data science team problem: inconsistent code that only the author can maintain. Standardized ETL patterns, schema validation, and experiment tracking transform individual notebooks into maintainable team infrastructure.'

Bridging Notebooks and Production with AI Rules

The notebook-to-production gap: Jupyter notebooks encourage exploration (run cells in any order, redefine variables, display inline). Production requires structure (deterministic execution, typed functions, proper imports). AI rules: make notebook code production-ready from day one. Rule: 'All notebook functions must have type hints and docstrings. All data transformations must be extractable as standalone functions (no cell-dependent state).' The AI: generates notebook code that can be directly extracted into production modules without rewriting.

Configuration management: notebooks hardcode everything (file paths, model parameters, database URLs). Production: reads from configuration. AI rule: 'Use pydantic Settings for all configuration. No hardcoded paths, URLs, or credentials in code. Load from environment variables with typed defaults.' The AI: generates configuration-driven code even in notebooks. The data scientist: changes the config to switch between local and production environments. The code: stays identical across environments.

Testing data transforms: data scientists rarely test transforms because the notebook IS the test (run the cell, visually inspect). Production requires automated tests. AI rule: 'Every transform function must have a corresponding test with fixture data. Use pytest with conftest.py for shared fixtures. Test edge cases: empty DataFrames, null values, schema changes.' The AI: generates test files alongside transform functions. The data scientist: has tests from the first commit, not added retroactively before a production deadline. AI rule: 'Notebooks and production are not two different codebases. They are one codebase at two maturity stages. AI rules: ensure the early stage already has the patterns the later stage requires. The gap narrows to zero.'

AI Rules for Machine Learning Workflows

Model serving consistency: each data scientist deploys models differently (Flask API, FastAPI, AWS Lambda, direct file load). AI rule: 'All model serving uses FastAPI with the standard predict endpoint pattern: POST /predict with pydantic request/response models. Use the ModelServer base class from ml_utils.' The AI: generates consistent serving code. The ML engineer: deploys any model using the same infrastructure. The DevOps team: maintains one deployment pattern instead of five.

Feature engineering standards: feature code is the most rewritten code in data science. AI rule: 'Feature functions are pure (no side effects), typed (input DataFrame, output DataFrame), and registered in the feature registry. Name pattern: compute_<feature_name>(df: pl.DataFrame) returning pl.DataFrame.' The AI: generates feature functions that are reusable, testable, and discoverable. The data scientist: builds on existing features instead of rewriting them.

Model versioning and reproducibility: AI rule: 'All models are versioned with DVC. Training scripts pin random seeds, log the full dependency environment, and store the training data hash. Model artifacts include: model file, config, metrics, and data hash.' The AI: generates reproducible training pipelines by default. The data scientist: can reproduce any previous result. The audit team: can trace any prediction to its training data. AI rule: 'ML workflows have more moving parts than traditional software: data, features, models, hyperparameters, environments. AI rules: standardize each moving part so the complexity is managed through conventions, not heroic individual effort.'

Data Scientist Quick Reference for AI Coding

Quick reference for data scientists using AI coding tools.

• Core benefit: AI rules bridge the notebook-to-production gap by generating production-ready patterns in exploration code
• Pipeline standards: consistent ETL patterns (extract/transform/load functions) across the entire data team
• Schema validation: rules enforce pandera/pydantic validation on all pipeline inputs — no silent data failures
• Experiment tracking: MLflow tracking added to every training script by default through rules
• Configuration: pydantic Settings for all config — no hardcoded paths, URLs, or credentials in code
• Testing: rules generate test files alongside transform functions — tests from day one, not retrofitted
• Model serving: consistent FastAPI predict endpoints — one deployment pattern for all models
• Feature engineering: pure, typed, registered functions — reusable across projects and team members