Data Scientist Interview Questions — Python

The loop body is not the cost. The machinery around it is.

For each element, CPython dispatches bytecode, checks the operand types at runtime, allocates a new object for the result, and updates reference counts. A Python integer is not four bytes in a row; it is an object with a header, scattered on the heap. Ten lakh iterations pay all of that ten lakh times.

Compiled libraries store the same data as a contiguous block of typed memory and run the loop in C, checking the type once for the whole array rather than once per element. The arithmetic was never the bottleneck; the per-element bookkeeping was.

# per-element interpreter overhead
scaled = []
for r in rainfall_mm:
    scaled.append(r * 0.0393)

# one call, loop runs in C
scaled = rainfall_arr * 0.0393

A list comprehension is faster than the explicit loop, because it avoids repeated attribute lookup on append and has a tighter bytecode path, but it is still an interpreted loop over boxed objects. Do not present it as vectorisation.

Two things worth saying in an interview.

Vectorisation only applies when the operation is uniform across elements. Anything genuinely sequential, where element n depends on the result for n-1, cannot be expressed that way, and the answer is then Numba, Cython, or accepting the loop.

And the trade-off is memory. A vectorised operation typically allocates a full output array, so a chained expression over a large array can allocate several intermediates. A generator loop uses almost nothing. On data that fits comfortably, take the speed; near the memory ceiling, chunk it.

Below a few thousand elements, the difference is microseconds and not worth restructuring code for. Measure before rewriting.

The convention most teams follow is about meaning, not performance. A list holds many of the same thing and its length varies. A tuple holds a fixed set of fields where position carries meaning.

sensor_ids = ["T-01", "T-02", "T-03"]      # list: many of one thing
reading = ("T-01", 34.2, "2026-08-19")     # tuple: one record

The concrete capability immutability gives you is hashability. A tuple of immutable values can be a dictionary key or a set member; a list cannot.

daily_max = {}
daily_max[("T-01", "2026-08-19")] = 41.7   # fine

daily_max[["T-01", "2026-08-19"]] = 41.7
# TypeError: unhashable type: 'list'

That single property is why tuples turn up as composite keys in caches, lookup tables and grouping logic all over real codebases.

The performance argument is usually overstated. Tuples are marginally smaller and construct slightly faster, and unless you are creating crores of them in a hot loop, that difference will not be what makes your job slow. Do not lead with it in an interview; lead with intent and hashability.

Once a tuple has more than three fields, positional access becomes a liability. reading[2] tells a reader nothing. NamedTuple or a frozen dataclass keeps the immutability and adds names:

from typing import NamedTuple

class Reading(NamedTuple):
    sensor_id: str
    celsius: float
    taken_on: str

Still hashable, still immutable, and r.celsius survives someone reordering the fields.

settings = {"retries": 3, "timeout": None}

settings["region"]              # KeyError: 'region'
settings.get("region")          # None
settings.get("region", "apac")  # 'apac'

Choosing between them is a decision about whether absence is normal. Reading a required configuration key with .get() and a silent default means a typo in the config file produces a working program with wrong behaviour, which is far harder to debug than a KeyError on startup. Reading an optional field from a parsed API response with brackets means one missing attribute crashes the whole batch.

The trap in that example is timeout. Both a missing key and a key set to None make .get() return None, so you cannot tell them apart:

settings.get("timeout")        # None — but the key exists
settings.get("nonsense")       # None — key doesn't exist

"timeout" in settings          # True

When the difference matters, test with in rather than inferring from the returned value. This bites when None is a legitimate stored value meaning “no limit” and your code treats it as “not configured” and substitutes a default.

One more thing people get wrong: .get() never inserts. If you want to read-or-create in one step, that is .setdefault().

by_city = {}
by_city.setdefault("Pune", []).append("order-118")

For accumulating into a dict repeatedly, collections.defaultdict(list) reads better and avoids repeating the default on every line.

import copy

template = [["mon", 0], ["tue", 0]]

shallow = copy.copy(template)      # or template[:] or list(template)
shallow[0][1] = 9
print(template)     # [['mon', 9], ['tue', 0]]  ← changed

deep = copy.deepcopy(template)
deep[1][1] = 5
print(template)     # [['mon', 9], ['tue', 0]]  ← safe

The shallow copy did create a new outer list. What it did not do is create new inner lists, so shallow[0] and template[0] are the same object, and mutating through either label is visible through both.

This turns up most often with a default structure reused across iterations. A roster template copied per employee, where every employee ends up sharing one shift list, produces a bug where updating one person’s schedule silently updates everyone’s. It looks like a database problem until you find the copy.

Deep copy is not free. It walks the entire object graph and reconstructs it, which on a large nested structure inside a loop is genuinely slow. Two alternatives worth mentioning. Rebuild the structure explicitly with a comprehension when you know its shape:

fresh = [[day, count] for day, count in template]

Or avoid the problem by making the inner elements immutable, since a tuple cannot be mutated through a shared reference at all.

Two details for the follow-ups. deepcopy handles cyclic references correctly by keeping a memo of objects it has already copied, so it does not recurse forever. And a class can control what happens by defining __deepcopy__, which matters for objects holding a database connection or a file handle that should not be duplicated.

The lock protects CPython’s internal state, principally reference counts. Without it, two threads incrementing the same object’s refcount concurrently would corrupt it, and making every refcount operation atomic would slow down single-threaded code, which is the overwhelming majority of Python code.

The consequence is blunt. Four threads doing arithmetic on four cores do not run four times faster; they take turns holding the lock and finish in roughly the time one thread would take, plus switching overhead.

Three things people get wrong.

It is not the language. It is a CPython implementation detail. Jython and IronPython have no GIL, and PyPy has one for the same reasons CPython does. Saying “Python has a GIL” is loose; saying “CPython has one” is correct and interviewers notice.

It is released during I/O. A thread waiting on a socket or a disk read gives up the lock, so other threads run. That is why threading genuinely helps for a hundred concurrent API calls despite the GIL.

C extensions can release it. Long-running numeric operations in numpy, scipy and similar libraries release the GIL around the compiled loop, so those genuinely use multiple cores while Python-level code cannot. A workload that spends its time inside such calls is much less GIL-bound than it first appears.

The escape routes for CPU-bound Python are processes, which each have their own interpreter and lock, or pushing the hot loop into compiled code through Cython or Numba.

One current detail worth stating precisely: Python 3.13 introduced an optional free-threaded build with the GIL disabled, following PEP 703. It is opt-in, not the default interpreter, carries a single-threaded performance cost, and the ecosystem is still catching up. Mentioning it accurately, with those caveats, reads better than either ignoring it or claiming the GIL is gone.

Same syntax, two directions. In the definition it packs; at the call it unpacks.

def log_event(event, *tags, **fields):
    print(event, tags, fields)

log_event("upload", "batch", "s3", size=940, retries=2)
# upload ('batch', 's3') {'size': 940, 'retries': 2}

payload = {"size": 940, "retries": 2}
log_event("upload", "batch", **payload)   # unpacking

The names are convention only; *a and **kw work identically. Stick with args and kwargs because everyone reading your code expects them.

Where these earn their place is a wrapper that must forward whatever it receives without knowing the signature it is wrapping. A retry helper, a timing wrapper, a decorator: all of them accept *args, **kwargs and pass them straight through, so they work with any function.

def with_retry(fn, *args, **kwargs):
    for attempt in range(3):
        try:
            return fn(*args, **kwargs)
        except TimeoutError:
            continue
    raise

Two things worth knowing beyond the basics.

A bare * forces everything after it to be keyword-only, which is a good way to stop callers passing a run of unlabelled booleans:

def export(path, *, overwrite=False, compress=True):
    ...
export("out.csv", overwrite=True)   # positional not allowed

And the cost of **kwargs is that it destroys the signature. Callers get no autocomplete, typos become silent extra dict entries rather than errors, and type checkers cannot help. Use it where forwarding is genuinely the point, not to avoid deciding what parameters a function takes.

def collect(row, batch=[]):
    batch.append(row)
    return batch

collect("r1")    # ['r1']
collect("r2")    # ['r1', 'r2']  ← not ['r2']
collect("r3")    # ['r1', 'r2', 'r3']

The list is created once, at def time, and stored on the function object itself. You can see it directly:

collect.__defaults__      # (['r1', 'r2', 'r3'],)

Every call that omits the argument gets that same list, so state leaks between calls that have nothing to do with each other.

The correct form is a sentinel:

def collect(row, batch=None):
    if batch is None:
        batch = []
    batch.append(row)
    return batch

is None rather than if not batch, because an empty list passed deliberately by the caller is falsy and would be silently replaced.

In production this rarely announces itself. A parsing helper with errors={} accumulates every error from every file processed since the service started, and memory grows steadily while each individual call looks correct. In a long-running worker that is a slow leak nobody attributes to the function.

The same evaluate-once rule catches other defaults. def report(as_of=datetime.now()) freezes the timestamp at import time, so a service running for a week reports the same date forever. Pass None and compute inside.

Immutable defaults are safe, since there is nothing to mutate: def f(n=0, label="") behaves as everyone expects.

There is one deliberate use of the behaviour, worth knowing but not worth writing: attaching a mutable default as a cheap per-function cache. functools.lru_cache does the same job without surprising the next reader.

staff = [
    {"name": "Riya", "dept": "ops", "joined": "2021-04-01"},
    {"name": "Arun", "dept": "eng", "joined": "2023-07-15"},
    {"name": "Meera", "dept": "ops", "joined": "2019-01-09"},
]

by_dept = sorted(staff, key=lambda r: (r["dept"], r["joined"]))

A tuple key sorts by the first element and uses the second to break ties, which handles most multi-field cases in one call.

Mixed directions are where stability earns its keep. You cannot put reverse=True on just one element of a tuple key, so you sort twice, least significant first:

rows = sorted(staff, key=lambda r: r["joined"], reverse=True)
rows = sorted(rows, key=lambda r: r["dept"])

Because Python’s sort is stable, the second sort preserves the date ordering within each department. Reverse the order of those two statements and the result is wrong. For numeric fields there is a shortcut: negate the value in the key, so key=lambda r: (r["dept"], -r["salary"]).

operator.itemgetter is often faster and clearer than a lambda for straightforward field access:

from operator import itemgetter
sorted(staff, key=itemgetter("dept", "joined"))

Two things to keep straight. sorted() returns a new list and works on any iterable; .sort() mutates a list in place and returns None. Writing rows = rows.sort() silently sets rows to None, and the failure surfaces later as a confusing TypeError.

And the key function is called exactly once per element, then the results are compared. So an expensive key such as a date parse costs n calls rather than n log n, which is the reason key= replaced the old comparison-function style.

books = [("Ghachar Ghochar", 2013), ("Em and the Big Hoom", 2012)]

titles  = [t for t, _ in books]                # list
by_year = {t: y for t, y in books}             # dict
years   = {y for _, y in books}                # set

Adding a condition filters, and a conditional expression before the for transforms:

recent = [t for t, y in books if y >= 2013]
tags = ["new" if y >= 2013 else "old" for _, y in books]

Those two ifs sit in different positions and do different jobs. Filtering goes after the for; choosing between two values goes before it. Getting them confused is a common syntax error.

Nesting reads in statement order, outer loop first:

shelves = [["a1", "a2"], ["b1"]]
flat = [code for shelf in shelves for code in shelf]

Read it as for shelf in shelves: then for code in shelf:, which is exactly the order you would write the loops. People frequently expect the reverse and get an error about an undefined name.

Where to stop is a judgement worth voicing. Two loops with a condition is around the limit before a comprehension stops being readable, and a reader having to decode it costs more than the two extra lines a loop would take. There is no rule here, only the next person reading it.

One memory point. Brackets build the entire list in memory before returning. Parentheses produce a generator expression instead, which yields lazily and is what you want when feeding a large sequence into sum or a loop:

total = sum(y for _, y in books)     # no intermediate list

Comprehensions have their own scope from Python 3 onwards, so the loop variable does not leak out.

def parse_log(path):
    with open(path) as f:
        for line in f:
            if line.startswith("ERROR"):
                yield line.strip()

for entry in parse_log("app.log"):   # one line in memory
    handle(entry)

The list version builds every matching line before the caller sees the first one. On a 40 GB log file that is the difference between a job that runs and one that gets killed by the memory limit.

Calling parse_log("app.log") executes nothing. It returns a generator object. The body runs only when something iterates it, advancing to the next yield and freezing there with all locals intact.

What you give up matters as much as what you gain.

gen = parse_log("app.log")
len(gen)          # TypeError
gen[0]            # TypeError
list(gen)         # works, but defeats the purpose

for x in gen: ... # consumes it
for x in gen: ... # yields nothing, silently

That last one is the failure that reaches production. A generator passed to two functions works for the first and gives the second an empty sequence, with no error and no warning. The downstream count comes out as zero and someone spends an afternoon on it. If two passes are needed, materialise once with list() and accept the memory cost, or call the generator function again.

One subtlety with the file example: the with block stays open while the generator is alive, and if the caller abandons it partway, the file is closed when the generator is garbage collected rather than at a predictable point.

yield from delegates to another iterable, which flattens nested generators without an explicit loop and correctly forwards values sent into the generator.

try:
    conn = connect(dsn)
except ConnectionError as exc:
    log.error("could not reach vendor API: %s", exc)
    raise
else:
    payload = conn.fetch_invoices()   # only if connect worked
finally:
    conn.close() if conn else None

The else block is the part most people never use, and it is what keeps the try narrow. Put fetch_invoices() inside the try and a ConnectionError raised deep inside the fetch gets caught by a handler written for the connection step, and your log says the wrong thing about the wrong operation.

finally runs whether or not an exception occurred, and whether or not the block returned. That makes it the right place for releasing a lock, closing a handle, or restoring a setting. In most cases a with block expresses the same intent more clearly, since the context manager owns the cleanup.

Three things that separate a working answer from a careful one.

Catch narrowly. A bare except: catches KeyboardInterrupt and SystemExit as well, so your job cannot be stopped cleanly. except Exception is better and still too broad as a default, since a typo raising NameError is swallowed and reported as a vendor failure.

Never swallow silently. except Exception: pass is how a nightly pipeline reports success while writing nothing. If you catch it, either handle it meaningfully or log it and re-raise.

Preserve the chain. When you wrap an error in your own exception type, raise LoadError("vendor feed failed") from exc keeps the original traceback attached. Without from, the underlying cause is harder to find in the logs.

One sharp edge worth knowing: a return inside finally discards any exception currently propagating, so an error disappears with no trace.

The guarantee is the point. A try/finally gives you the same thing, and with moves the cleanup logic into the object that owns the resource, so every caller gets it right without remembering.

class Stopwatch:
    def __init__(self, label):
        self.label = label

    def __enter__(self):
        self.start = time.perf_counter()
        return self          # bound to `as`

    def __exit__(self, exc_type, exc, tb):
        elapsed = time.perf_counter() - self.start
        log.info("%s took %.2fs", self.label, elapsed)
        return False         # do not suppress

with Stopwatch("model export") as sw:
    export_weights()

__exit__ receives three arguments describing any exception that escaped the block, all None if the block finished cleanly. Returning a truthy value from __exit__ suppresses that exception, which is occasionally what you want and much more often a mistake. Return False, or nothing at all, unless you deliberately mean to swallow it.

Whatever __enter__ returns is what as binds. Returning self is the common choice; returning None is fine when the block does not need a handle.

For most cases the generator form is shorter and reads better:

from contextlib import contextmanager

@contextmanager
def temp_setting(obj, **overrides):
    saved = {k: getattr(obj, k) for k in overrides}
    for k, v in overrides.items():
        setattr(obj, k, v)
    try:
        yield obj
    finally:
        for k, v in saved.items():
            setattr(obj, k, v)

The try/finally around the yield is essential. Without it, an exception in the body propagates out of the yield and the restoration never runs, which is exactly the guarantee you were trying to provide.

One gotcha: opening several resources with a single with and commas is fine, but if the second one fails during setup, the first is still cleaned up correctly. Nesting manually with plain assignments does not give you that.

class Turbine:
    site = "Jaisalmer"      # class attribute, shared
    log = []                # class attribute, shared and mutable

    def __init__(self, tag):
        self.tag = tag      # instance attribute

Lookup goes instance first, then class. So t.site finds nothing on the instance and falls back to the class, and every turbine reports the same site. That is useful for genuine constants and defaults.

Assignment does not follow the same path. Writing t.site = "Bhuj" creates an instance attribute that shadows the class one for that object only; the class attribute is untouched and every other instance still sees Jaisalmer. Reading falls back, writing does not.

The mutable class attribute is the real trap:

a, b = Turbine("T1"), Turbine("T2")
a.log.append("fault")
b.log              # ['fault']

a.log.append(...) never assigns, so it never creates an instance attribute. It reaches through to the shared list and mutates it. Two objects that should be independent now share state, and in a long-running service that list also grows without bound. Anything mutable belongs in __init__ as self.log = [].

self is not a keyword, just a convention. Python passes the instance as the first positional argument to any method called on it, so t.spin() is Turbine.spin(t). Rename it and the code works and every reviewer will object.

One detail for the follow-up: __init__ initialises an object that already exists and must return None. The object is created by __new__, which you rarely touch outside of immutable subclasses and singletons.

The problem is a single shared namespace. One project needs an older release of a plotting library that a colleague’s project cannot use, and there is exactly one place to install it.

python -m venv .venv
source .venv/bin/activate      # Windows: .venvScriptsactivate
pip install -r requirements.txt

venv is in the standard library from Python 3.3 onwards, so nothing needs installing first. Activating it puts that environment’s python and pip at the front of your PATH, so installs land inside the project directory rather than in the system interpreter.

Two failures make this concrete.

Installing into the system Python on Linux can break OS tooling that depends on specific package versions. Recent versions of Debian and Ubuntu now refuse a bare pip install outside a virtual environment for exactly this reason, and the error message confuses people who have never used one.

The subtler one is drift. Code runs on your machine and fails in the pipeline, because your machine picked up a library version installed eighteen months ago for something else. Nobody can reproduce it and nobody can explain it. A per-project environment plus a pinned requirements file removes the whole category.

Pin what you install. pip freeze > requirements.txt captures exact versions including transitive dependencies, which is what you want for reproducing a deployment. A hand-written file listing only your direct dependencies with ranges is better for a library others will install. Tools like Poetry and uv manage both sides with a lock file, and are worth naming if the interviewer asks what you use.

One practical note: keep .venv/ out of version control and commit the requirements file instead. The environment is derived; the specification is the artefact.

The syntax hides an assignment. These two are identical:

@audit
def issue_refund(ticket_id, amount):
    ...

def issue_refund(ticket_id, amount):
    ...
issue_refund = audit(issue_refund)

That second form is the whole concept. audit receives the original function object and returns something else, and the name now points at the replacement.

Writing one:

import functools

def audit(fn):
    @functools.wraps(fn)
    def wrapper(*args, **kwargs):
        log.info("calling %s", fn.__name__)
        result = fn(*args, **kwargs)
        log.info("%s returned", fn.__name__)
        return result
    return wrapper

Three details carry the weight. *args, **kwargs means the wrapper forwards anything, so one decorator works on functions with any signature. return result is easy to omit and turns every decorated function into one that returns None, which is a bug that passes import and fails at runtime. And functools.wraps copies the original’s __name__, __doc__ and __wrapped__ onto the wrapper, without which tracebacks, help output and anything doing introspection all report wrapper.

A decorator taking arguments needs one more layer, because @retry(times=3) calls retry(times=3) first and applies the result as the decorator:

def retry(times):
    def decorator(fn):
        @functools.wraps(fn)
        def wrapper(*args, **kwargs):
            for _ in range(times - 1):
                try:
                    return fn(*args, **kwargs)
                except TimeoutError:
                    continue
            return fn(*args, **kwargs)
        return wrapper
    return decorator

Three nested functions is the standard shape and the part people fumble under pressure. Practise writing it.

Two things worth saying. Decorators run at import time, when the def is executed, so an expensive decorator body costs you on every import. And methods work unchanged: self arrives as the first element of args, since by then it is an ordinary positional argument.

import timeit

setup = "from ranking import score_v1, score_v2, sample"
t1 = timeit.timeit("score_v1(sample)", setup=setup, number=200)
t2 = timeit.timeit("score_v2(sample)", setup=setup, number=200)
print(f"v1 {t1:.3f}s  v2 {t2:.3f}s  ratio {t1/t2:.2f}x")

In a notebook, %timeit does the same thing with automatic repeat selection and reports the spread.

Three things make a benchmark honest.

Repeat enough. A single run of a fast function measures scheduler noise more than your code. timeit loops and reports the total, which averages the jitter out.

Report the minimum across repeats, not the mean. External interference can only ever make a run slower, never faster, so the fastest observed run is the closest estimate of the true cost. timeit.repeat gives you several totals to take the minimum from.

Keep setup out of the measurement. Building the input inside the timed statement measures the construction, and if both versions build it, the real difference gets diluted into noise.

The failure that produces wrong conclusions: measuring on a 100-element list. Fixed costs such as function call overhead dominate at that size, and the version that wins there frequently loses at ten lakh elements where allocation behaviour matters. Benchmark at the size you actually run.

Cache effects also mislead. If the first version populated a cache or warmed the CPU’s data cache, the second looks faster for reasons unrelated to the code. Run them in both orders and check the ordering does not change the answer.

For I/O-bound work, timeit is the wrong tool entirely, since you would be measuring the network. Profile the wall-clock time across many runs and look at the distribution, because the tail matters more than the average there.

The difference shows up as soon as the collection grows.

blocked = set(load_blocked_pincodes())    # 90,000 entries

if pincode in blocked:                    # average O(1)
    reject(pincode)

With a list, in walks the elements one by one until it finds a match, so checking one lakh incoming pincodes against a 90,000-entry list is roughly nine billion comparisons. Against a set it is one lakh hash lookups.

Set operations replace loops you would otherwise write by hand:

served = {"560001", "560034", "560076"}
requested = {"560034", "411001"}

requested & served      # {'560034'}   both
requested - served      # {'411001'}   requested but not served
requested | served      # union
requested ^ served      # in one but not both

Three caveats worth raising before the interviewer does.

The O(1) is average case. Hash collisions degrade lookups, and in pathological cases where many elements hash to the same bucket, behaviour approaches linear. For ordinary strings and integers this is not something you will meet, but the honest phrasing is “average O(1)”, not “always O(1)”.

Only hashable objects can go in. A set of lists raises TypeError; a set of tuples is fine.

Order is not preserved in any way you should rely on. set(names) deduplicates and returns the items in hash order, which will surprise anyone expecting the original sequence. If order matters, list(dict.fromkeys(names)) deduplicates while keeping first occurrence, since dicts preserve insertion order from Python 3.7 onwards.

def load(paths):
    for p in paths:
        with open(p) as f:
            for line in f:
                yield json.loads(line)

def drop_incomplete(docs):
    for d in docs:
        if d.get("abstract") and d.get("year"):
            yield d

def tokenise(docs):
    for d in docs:
        d["tokens"] = d["abstract"].lower().split()
        yield d

def to_features(docs, vocab):
    for d in docs:
        yield {t: d["tokens"].count(t) for t in vocab}

pipeline = to_features(tokenise(drop_incomplete(load(paths))), vocab)

Constructing pipeline runs nothing. Only when something iterates it does one record get pulled through all four stages, so peak memory is one record plus the vocabulary.

Three properties make this worth the structure.

Each stage is a function over an iterable, so testing it needs no files: list(drop_incomplete([{"abstract": "x"}])). That is far easier than testing a single function that does all four things.

Stages compose in any order and a new one slots in without touching the others.

And filtering early is free. drop_incomplete sitting before tokenise means discarded records never get tokenised, and the saving is automatic rather than something you had to arrange.

Two things that break the model.

Calling list() anywhere in the chain materialises everything, and you are back to holding the dataset. That includes an innocent-looking len(), which forces the same thing.

And any stage needing global knowledge cannot be lazy. Building the vocabulary, computing a mean for standardisation, or sorting all require a full pass. The usual answer is two passes: one to compute the statistic, one to stream the transformation using it. Fitting a scaler on the streaming pass would use future records to transform earlier ones, which is leakage.

For counting drops, keep a counter in an outer scope and increment inside the stage, since generator locals survive across yields.

lanes = ["A", "B", "C", "D", "E", "F"]

lanes[1:4]      # ['B', 'C', 'D']   stop is exclusive
lanes[:3]       # ['A', 'B', 'C']
lanes[-2:]      # ['E', 'F']        last two
lanes[::2]      # ['A', 'C', 'E']   every second
lanes[::-1]     # ['F', 'E', ...]   reversed

The exclusive stop is what makes lanes[:3] + lanes[3:] reconstruct the original with no overlap and no gap, which is the reason for the convention.

Slicing produces a new list. That is why copy = original[:] is a common idiom for making a copy:

toll_lanes = lanes[:]
toll_lanes.append("G")
print(len(lanes))     # 6 — unaffected

The copy is shallow, though. The new list is independent, and the objects inside it are the same objects. With a list of lists, mutating an inner list is visible through both.

Slices behave differently from indexes at the boundaries, which is a genuine gotcha:

lanes[10]       # IndexError
lanes[3:99]     # ['D', 'E', 'F'] — no error
lanes[8:12]     # [] — no error

An out-of-range slice clamps silently. That is convenient in a paging loop and dangerous when it hides an off-by-one, since a wrong index gives you an empty list rather than an exception.

Two practical notes. [::-1] reverses a copy while list.reverse() reverses in place, so pick by whether you need the original. And on a very large list, slicing copies every element, so taking big[:-1] inside a loop is quietly quadratic; itertools.islice iterates without copying.

from functools import lru_cache

@lru_cache(maxsize=512)
def embed_phrase(phrase: str) -> tuple[float, ...]:
    return tuple(model.encode(phrase))

embed_phrase.cache_info()
# CacheInfo(hits=8140, misses=512, maxsize=512, currsize=512)

cache_info() is the first thing to check after adding one. A hit rate near zero means the arguments vary more than you assumed and you are paying the caching overhead for nothing.

Three conditions have to hold.

Arguments must be hashable. A list or a dict raises TypeError: unhashable type. Convert to a tuple or a frozenset at the call site, which is why the return above is a tuple rather than a list.

The function must be pure. Same input, same output, no side effects. Caching a function that reads a file or the current time returns the first result forever, and the bug looks like stale data with no obvious cause.

The return value must be treated as immutable. The cache stores the object, not a copy, so if the caller mutates a returned list, every subsequent cache hit gets the mutated version. Returning tuples avoids the whole problem.

maxsize is the setting that matters in production. lru_cache(maxsize=None), and its shorthand functools.cache from Python 3.9, never evict, so a service caching on a wide key space grows until it is killed. Bound it unless the key space is genuinely small and fixed.

Two extras. The cache is per process, so eight worker processes hold eight separate caches with no sharing. And decorating a method caches on self too, which keeps the instance alive for as long as the entry exists; that is a real memory leak in a long-lived process, and cached_property or an explicit instance-level cache is the safer pattern there.

The functions that come up most in analysis work:

from itertools import chain, islice, product, combinations

# treat several files as one stream
records = chain(load(a), load(b), load(c))

# first 500 without building the rest
head = list(islice(records, 500))

# every hyperparameter combination
grid = product([0.01, 0.1], [16, 64, 256], ["adam", "sgd"])
for lr, batch, opt in grid:
    run_trial(lr, batch, opt)

# every unordered pair of features
for a, b in combinations(feature_names, 2):
    check_correlation(a, b)

product replaces three nested loops and reads better. combinations gives unordered pairs without repetition, so (a, b) appears and (b, a) does not, which is what you want for a symmetric measure like correlation. permutations gives both orders and is what you want when order carries meaning.

Two things about laziness.

chain does not concatenate. It yields from each source in turn, so joining a hundred large files costs nothing in memory.

islice slices any iterable, including an infinite generator, and unlike [:500] it does not require a sequence. It also cannot take negative indices, since it never knows where the end is.

The gotcha everyone meets once: these return iterators, consumed after a single pass.

grid = product([1, 2], [3, 4])
len(list(grid))    # 4
len(list(grid))    # 0  ← exhausted

A parameter grid iterated in a loop and then printed for the log comes out empty, with no error. Wrap in list() if you need it twice.

Two more worth naming. itertools.groupby groups only consecutive equal keys, so sort first. And count with islice generates a bounded run from an unbounded source, which is the usual way to sample from something infinite.

a = 256
b = 256
a == b      # True
a is b      # True

x = 1000
y = 1000
x == y      # True
x is y      # False

Nothing changed about the language between those two blocks. CPython pre-creates integer objects from -5 to 256 and reuses them, so both names point at the identical cached object. At 1000 there is no cache, so two separate objects exist with equal value.

Short strings behave similarly through interning, and the rules for when interning happens vary by how the string was created and by CPython version. Both behaviours are implementation details of CPython, not guarantees of Python, and code that depends on them can break on another interpreter or another release.

The rule that follows: use == for value, and is only where identity is genuinely the question. In practice that means the singletons.

if config is None:          # correct
if config == None:          # works, but non-idiomatic

is None is preferred for a real reason beyond style. == calls the left operand’s __eq__, and a class can define that method to return anything it likes, including True when compared with None. is cannot be overridden, so it always answers the actual question.

The failure this causes in production is a comparison that passes every test on small values and fails on real data. A validation checking status_code is 200 works throughout development and breaks when someone loads the code path where the value is parsed from JSON. It reads as a mysterious intermittent bug, and static analysis tools now flag it precisely because it is so common.

The falsy set is short and worth knowing exactly: None, False, zero of any numeric type, empty sequences and mappings ("", [], (), {}, set()), and any object whose class defines __bool__ or __len__ to say so. Everything else is truthy.

The bug this creates is specific and common:

def apply_discount(order_total, override=None):
    if not override:
        override = DEFAULT_DISCOUNT
    return order_total * (1 - override)

apply_discount(2000, override=0)   # applies DEFAULT, not 0

A caller who explicitly asked for zero discount gets the default instead, because 0 is falsy. Nobody notices until a customer is charged a discount that was meant to be waived. The fix is to test for the thing you actually mean:

if override is None:
    override = DEFAULT_DISCOUNT

The same shape appears with empty strings, where a deliberately blank field is replaced by a placeholder, and with empty lists, where “query returned no rows” is confused with “query was never run”.

That said, if items: is idiomatic and correct when you genuinely mean “is this container non-empty”. Prefer it over if len(items) > 0:, which says the same thing more loudly. The judgement is about whether zero and empty are meaningful values in your domain.

One related behaviour to have ready. or and and return one of their operands, not a boolean:

name = user_input or "guest"      # 'guest' if user_input is falsy

Convenient, and it carries the same zero-and-empty-string trap.

One question decides it: while this code runs, is the CPU busy or waiting?

For simulating five hundred parameter settings, each running a numeric fit for thirty seconds, processes are the answer. The work is Python-level computation and only separate interpreters get separate cores.

For fetching two thousand documents, asyncio or threads. Both overlap the waiting. Threads need no rewrite of the calling code, which matters when the client library is synchronous. Asyncio scales further because a coroutine costs far less than a thread’s stack, but it demands the whole path be async, and one blocking call inside a coroutine stalls everything.

For a mixed workload, which is the common real case, layer them. Fetch concurrently with threads or asyncio, then hand the CPU-heavy parsing to a process pool. Trying to do both with one mechanism means one half of the job is on the wrong tool.

Two costs worth naming. A hundred processes means a hundred interpreters and a hundred copies of whatever each loads, which on a machine with 16 GB is often the binding constraint rather than cores. And anything crossing a process boundary is pickled, so sending a large object to a worker that computes briefly can cost more than doing the work inline.

Only threads and asyncio share Python objects directly. That is convenient and it is also why threads need locks around mutable shared state, which processes do not.

from multiprocessing import Pool

def fit_one(seed):
    return run_simulation(seed, n_steps=50_000)

if __name__ == "__main__":
    with Pool(processes=8) as pool:
        results = pool.map(fit_one, range(200))

The main guard is not optional. On Windows and on macOS with the spawn start method, each child re-imports the main module, and without the guard each child runs the pool creation again, spawning more children until the machine gives up.

Three costs shape when this is worth it.

Startup. Spawning eight interpreters takes a noticeable fraction of a second each, and with spawn each child re-imports your modules. For work measured in milliseconds per item, startup dominates and the parallel version is slower.

Serialisation. Arguments and return values are pickled. A worker that takes a small seed and returns a small result is ideal. One that receives a large array, computes for fifty milliseconds and returns another large array spends most of its time in pickle.

Memory. Each worker is a full interpreter with its own copy of imported modules and any data it loads. Eight workers each loading a 2 GB lookup table is 16 GB.

The picklability constraint catches people. Lambdas, locally defined functions, open file handles and database connections all fail. The worker function must be importable at module level, and each worker builds its own connections.

Use imap_unordered when results can be consumed as they finish, which keeps memory flat over a long run instead of collecting everything. chunksize matters too: with many small tasks, the default sends them one at a time and the IPC overhead swamps the work.

For a large read-only table, initialise it once per worker with the pool’s initializer rather than passing it with every task.

from pathlib import Path

def load_trials(
    path: Path,
    min_score: float = 0.0,
) -> list[dict[str, float]]:
    ...

Python stores those annotations and ignores them. The function will happily return a string:

def count() -> int:
    return "seven"      # runs fine, mypy flags it

So the value comes entirely from tooling and from readers. Three concrete returns.

Errors before runtime. A function annotated -> dict | None whose caller indexes the result directly is a TypeError waiting for the one input that yields None. mypy finds it in seconds; a test suite finds it only if that case is covered.

Precision that comments cannot give. records: list[dict[str, float]] tells the next reader the exact shape. A docstring saying “takes records” does not, and drifts silently when the code changes.

Editor support. Autocomplete and rename-refactoring work far better with annotations, which is a daily saving rather than an occasional one.

Some libraries do read them at runtime. dataclasses builds __init__ from the annotations, and pydantic validates and coerces incoming data against them, which is why it is common at API and config boundaries.

Two practical points for research code. Annotate function signatures and skip local variables; the signatures carry nearly all the value at a fraction of the effort. And from __future__ import annotations lets you write list[dict] and X | None on older versions, since the built-in generics landed in 3.9 and the | union syntax in 3.10.

Adopt gradually. mypy checks what is annotated and leaves the rest alone, so a large notebook-grown codebase can be typed one module at a time rather than all at once.

LEGB is the lookup order, and the part that catches people is not the lookup but the assignment rule.

tax_rate = 0.18

def price_with_tax(base):
    print(tax_rate)          # UnboundLocalError
    tax_rate = 0.05
    return base * (1 + tax_rate)

The error appears on a line that only reads the variable. Python scans the whole function body at compile time, sees an assignment to tax_rate, and marks it local for the entire function, including lines above the assignment. At the moment of the print it is local and not yet bound.

global says the name refers to the module-level variable, and nonlocal says it refers to the nearest enclosing function’s variable:

def make_counter():
    count = 0
    def record():
        nonlocal count       # without this: UnboundLocalError
        count += 1
        return count
    return record

count += 1 is an assignment, so the same rule applies inside the inner function. nonlocal points it back out one level. global would not work here, since count is not at module level.

Two things worth saying in an interview. Only functions, classes, modules and comprehensions create scopes. for, if and with do not, so a loop variable is still visible after the loop ends, which surprises people coming from C-family languages.

And global in production code is usually a smell. Module-level mutable state makes functions untestable and behaviour order-dependent, and in a multi-threaded worker it becomes a race condition. Pass the value in and return the result instead.

from dataclasses import dataclass, field

@dataclass
class TrialResult:
    trial_id: str
    accuracy: float
    params: dict[str, float]
    notes: list[str] = field(default_factory=list)

r = TrialResult("t-04", 0.913, {"lr": 0.01})
r.accuracy          # attribute access, not r["accuracy"]

The generated __repr__ is worth the decorator on its own. Printing this shows every field with its value, where a plain class shows an object address.

The comparison that matters:

The typo row is the practical argument. result["acuracy"] = 0.9 on a dict creates a second key and the original stays untouched, so a metric silently stops updating. On a dataclass it raises immediately.

field(default_factory=list) is required for mutable defaults. Writing notes: list[str] = [] raises a ValueError at class definition time, because a single list would be shared across every instance. The dataclass machinery refuses rather than letting you create that bug.

@dataclass(frozen=True) makes instances immutable and generates a __hash__, so they can go in sets and be used as dict keys. For a record you never intend to modify, that is the safer default.

Choose a dict for genuinely dynamic data such as a parsed JSON payload of unknown shape. Choose a NamedTuple when it must be a tuple, for unpacking or for interoperating with tuple-based code. Choose a dataclass for anything else, and reach for pydantic when the data comes from outside and needs validating.

multipliers = []
for factor in [2, 5, 10]:
    multipliers.append(lambda x: x * factor)

[m(3) for m in multipliers]     # [30, 30, 30] ← not [6, 15, 30]

Nothing is captured at the moment each lambda is created. Each one holds a reference to the enclosing variable factor, and looks it up when called. By the time you call them, the loop has finished and factor is 10.

This is late binding, and it is the correct behaviour for closures generally. A closure that captured values would be useless for the counter pattern, where the whole point is that the inner function sees updates to the outer variable.

Two ways to bind the value at definition time. A default argument, which is evaluated once when the lambda is defined:

for factor in [2, 5, 10]:
    multipliers.append(lambda x, f=factor: x * f)

Or functools.partial, which stores the argument explicitly:

from functools import partial
from operator import mul

multipliers = [partial(mul, f) for f in [2, 5, 10]]

Where this bites in real work is callbacks. Registering a handler per queue inside a loop, or building a list of retry functions each bound to a different endpoint, and every one of them ends up pointing at the last item. The symptom is that all your workers process the same partition, which looks like a configuration problem rather than a language one.

A comprehension has exactly the same issue, since it also builds the functions before any of them run. And each closure holds its variables in a cell object, visible as fn.__closure__, which is worth knowing if an interviewer pushes on the mechanism.

Compare the same operation three ways on a list of ticket prices:

prices = ["450", "1200", "300"]

list(map(int, prices))                    # clean
list(map(lambda p: int(p) * 1.05, prices))
[int(p) * 1.05 for p in prices]           # clearer

The first is good code. map with an existing callable is compact and says exactly what it does. The second is where readability drops: a lambda inside a map means reading right to left through two layers to work out what happens to each item. The comprehension states the transformation first and the source second, which is how people read.

Filtering follows the same pattern:

list(filter(lambda p: int(p) > 400, prices))
[p for p in prices if int(p) > 400]

Combining both in one comprehension is natural and needs no nesting:

[int(p) * 1.05 for p in prices if int(p) > 400]

Two limits on lambdas that are worth stating. A lambda holds one expression, so no statements, no assignments, no try blocks. A conditional expression is allowed because it is an expression: lambda p: "high" if p > 1000 else "low".

And a lambda assigned to a name gains nothing over def, while losing a useful name in tracebacks. charge = lambda x: x * 1.05 shows up as <lambda> when it raises, and PEP 8 recommends def there.

One behavioural difference worth knowing: in Python 3, map and filter return lazy iterators, not lists. They consume nothing until iterated and are exhausted after one pass, so a map object you iterate twice gives you results the first time and nothing the second.

The notebook and the deployed system differ in four ways, and each one needs work.

Hidden state becomes explicit arguments. A cell using df_train defined eleven cells earlier works only because that variable is still in memory. Wrap it in a function taking what it needs and returning what it produces. That single change makes the code testable, reusable and reviewable.

def build_features(readings: list[dict],
                   vocab: dict[str, int]) -> list[list[float]]:
    ...

Paths and constants come from configuration. /home/ayan/Downloads/aug_export.csv and a threshold typed into a cell both have to become parameters read from arguments or a settings object.

Order becomes guaranteed. Cells can run in any order, and the engineer’s environment will run top to bottom exactly once. Restart the kernel and run everything before you hand anything over; if it fails, the notebook never described what produced your results.

Dependencies get pinned. An exported environment file, so the same versions install on the other side.

The technical risk to raise unprompted is training-serving skew. Preprocessing written twice, once in your notebook and once in the serving code, will drift, and the model then sees features it was not trained on. Ship one implementation that both paths import. If a scaler or an encoder was fitted on training data, its fitted state has to be serialised and shipped with the model, not refitted at serving time.

Hand over more than code: the training data version or query, the fitted artefacts, the metrics you measured and on which split, and the expected input schema with types and ranges. An engineer who cannot tell whether the model is behaving correctly cannot operate it.

A for loop is doing three things behind the scenes: call iter() on the object, call next() repeatedly, and stop when StopIteration is raised.

racks = ["R1", "R2"]
it = iter(racks)          # list -> list_iterator
next(it)                  # 'R1'
next(it)                  # 'R2'
next(it)                  # StopIteration

The list itself has no position. Each iter() call creates a new iterator with its own cursor, which is why two nested loops over the same list work independently.

An iterator returns itself from __iter__, which is what lets you pass one to a for loop directly. It also means iterating it consumes it. A file object is its own iterator, so a second loop over an already-read file yields nothing at all, with no error. That is the same silent-empty failure a consumed generator produces.

Writing one by hand:

class Countdown:
    def __init__(self, start):
        self.n = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.n <= 0:
            raise StopIteration
        self.n -= 1
        return self.n + 1

Making it self-consuming like this is a design choice with a cost: Countdown(3) can only be looped once. The better pattern for a reusable collection is to keep the state out of the container and return a fresh iterator, or simply make __iter__ a generator function, which gives you a new one per call for free.

Two details for follow-ups. StopIteration is caught by the for loop itself, so you never see it unless you call next() manually; pass a default to next(it, None) to avoid it. And iter() falls back to __getitem__ with integer indices from zero for classes predating the protocol, which is why some old sequence classes are iterable without defining __iter__.

crops = ["wheat", "gram", "mustard"]

for i, crop in enumerate(crops, start=1):
    print(f"Plot {i}: {crop}")

The start argument saves the i + 1 that otherwise appears everywhere in report-numbering code.

Zip pairs elements positionally:

plots = ["P1", "P2", "P3"]
yields = [22.5, 19.1]

list(zip(plots, yields))
# [('P1', 22.5), ('P2', 19.1)]  ← P3 vanishes

No error, no warning. The third plot is gone from the output, and if this is feeding a report, the total is short by one row and nothing indicates why. It is a genuinely common source of quietly wrong results when two sequences come from different sources and one is missing a record.

Two ways to guard against it. From Python 3.10, zip(plots, yields, strict=True) raises ValueError on a mismatch, which is what you want whenever equal length is an invariant. Before 3.10, assert the lengths, or use itertools.zip_longest when the shorter sequence should be padded:

from itertools import zip_longest
list(zip_longest(plots, yields, fillvalue=None))

Combining both is idiomatic and reads well:

for i, (plot, y) in enumerate(zip(plots, yields), start=1):
    ...

Note the parentheses around the inner pair. Without them the unpacking fails.

Two more things worth knowing. zip(*rows) transposes, turning rows into columns, and it is the standard way to unzip a list of pairs back into two tuples. And in Python 3, zip and enumerate return lazy iterators, so wrap in list() if you need to look at the result more than once.

from scipy import stats

stat, p = stats.ttest_ind(group_a, group_b, equal_var=False)
print(f"t = {stat:.3f}, p = {p:.4f}")

equal_var=False gives Welch’s t-test, which does not assume the two groups have equal variance. It is the safer default and loses very little power when the variances happen to be equal, so most practitioners now use it routinely.

What scipy will not do is stop you. Feed it two skewed distributions, or paired samples through an independent test, or data with wild outliers, and it returns a number with the same confidence as always. The function has no way to know your design.

So the checks come first. Are the observations independent, or is the same user in both groups? Is the sample large enough that the central limit theorem covers the non-normality? Are there outliers driving the difference, which for a t-test on the mean can flip the conclusion on their own?

Two things about interpreting the p-value.

It is not the probability the null is true, and it is not the probability the effect is real. It is the probability of data at least this extreme under the null. That distinction is what interviewers are usually listening for.

And with a large sample, a trivially small difference becomes statistically significant. At two lakh observations a 0.3% lift can produce p = 0.001 and be worth nothing commercially. Always report the effect size and a confidence interval next to the p-value, because those describe the magnitude that the p-value does not.

Multiple testing is the other trap. Running twenty comparisons at the 0.05 level gives roughly a 64% chance of at least one false positive. Apply a correction, or say clearly that the analysis was exploratory.

The non-parametric alternative is stats.mannwhitneyu when normality is genuinely doubtful.

import random, os
import numpy as np

SEED = 20260819
random.seed(SEED)
np.random.seed(SEED)
os.environ["PYTHONHASHSEED"] = str(SEED)

Seeding one and assuming the rest follow is the usual cause. A shuffle from random and a sample from numpy draw from entirely separate state.

The PYTHONHASHSEED line has a catch worth knowing: setting it inside the script is too late, because hash randomisation is fixed when the interpreter starts. It has to be in the environment before launch. Where it matters is any code iterating over a set of strings, since the order varies between processes and a downstream sample then differs.

Three other sources a seed does not touch.

Concurrency. Threads or processes completing in a different order change the order results are combined, and floating-point addition is not associative, so the totals differ in the last digits. Seeding each worker separately and combining deterministically is the fix.

GPU operations. Many are non-deterministic by default because of atomic accumulation order. Frameworks expose a deterministic mode, usually at a performance cost.

Library defaults. Scikit-learn estimators taking random_state use the global numpy state when you leave it as None, which makes the result depend on how many random draws happened earlier in the session.

The pattern that survives review is to stop using global state. Create an explicit generator and pass it where it is needed:

rng = np.random.default_rng(SEED)
idx = rng.choice(len(records), size=500, replace=False)

default_rng is the current numpy API and is preferred over np.random.seed, because the object is passed explicitly and no unrelated code can consume from it. Log the seed with every run, or you cannot reproduce even the runs that were deterministic.

while True:
    try:
        job = queue.pop()
        proces_job(job)          # typo
    except:
        log.warning("job failed, retrying")

Two things are broken here and neither is visible. The typo raises NameError on every single iteration, and the handler reports it as a job failure, so the log fills with retry messages about a queue that is working perfectly. And pressing Ctrl-C raises KeyboardInterrupt, which the bare except catches, so the loop keeps running and the only way out is to kill the process.

The exception hierarchy is why. KeyboardInterrupt, SystemExit and GeneratorExit inherit from BaseException directly, not from Exception, precisely so that ordinary handlers do not intercept them. A bare except: sits at the BaseException level and undoes that design.

except Exception:        # excludes Ctrl-C and sys.exit
except (TimeoutError, ConnectionError):   # better still

Catch what you can actually do something about. If the handler’s response is “retry the network call”, then catch the network errors, and let a KeyError from malformed data propagate to somewhere that knows what to do with it.

There are two legitimate uses of a broad catch. A top-level supervisor loop in a long-running worker, where the job must not die because one item failed, and a plugin boundary where you cannot know what user code raises. In both cases the handler must log the full traceback and re-raise or record the failure:

except Exception:
    log.exception("job %s failed", job.id)   # includes traceback
    failed.append(job.id)

log.exception rather than log.warning is the detail that matters. Without the traceback you have a message and no way to find the line.

Interviewers ask this because a swallowed exception is the hardest class of bug to debug: the program reports success while doing nothing.

pip install scikit-learn in March and again in September gives two different versions. Library defaults change between releases, algorithms get reimplemented, and a solver’s default tolerance shifts. Your code is identical and the numbers are not.

Two files with different jobs:

# requirements.in — what you asked for
scikit-learn>=1.4
scipy

# requirements.txt — what actually got installed
scikit-learn==1.4.2
scipy==1.13.0
joblib==1.4.2
threadpoolctl==3.5.0
numpy==1.26.4

The second is generated, by pip-compile from pip-tools or by pip freeze, and it includes the transitive dependencies. Those matter as much as the direct ones: a change in a numerical backend changes results even when every package you named stayed the same.

Pin the Python version as well. Behaviour differs between minor releases, and an environment built on 3.11 is not the same as one on 3.12.

Two limits worth stating honestly.

A lock file pins Python packages and not the system libraries beneath them. BLAS implementations and CPU instruction sets affect floating-point results in the last digits, which is usually irrelevant and occasionally is not. A container image pins the whole stack including the OS libraries, which is why it is the stronger answer for anything that must reproduce exactly.

And environment reproducibility does nothing about the data. If the source table has been corrected or extended since March, an identical environment gives a different answer for entirely legitimate reasons. Record the data version, the query, and the row count alongside the environment.

Conda users get the same with an exported environment file, which additionally captures non-Python dependencies. uv and Poetry both produce lock files with the same guarantee and better resolution behaviour.

The defining problem is hidden state. A variable defined in a cell you later edited still exists in memory, so a notebook can display correct output that its own code cannot reproduce. Restart-and-run-all is the only test that matters, and it should be habitual before sharing, not something you do once at the end.

Beyond that, five things.

Paths. /Users/ayan/Desktop/exp3.csv works nowhere else. Read a root from an environment variable or a config cell at the top.

Seeds at the top, in one place. Set them in the first executable cell so anyone can see what they are and change them deliberately.

No cells that must be skipped. A cell that downloads 40 GB and is normally commented out is a landmine. Either guard it with a flag or move it out of the notebook.

Nothing edited by hand. A value typed over a computed result is invisible in the code and gone the moment the cell reruns.

Pinned dependencies alongside. The environment file lives next to the notebook, not in someone’s memory.

Even after all that, a colleague can get different numbers if the underlying data has changed. Record the data version, the query and the row count in a cell so a mismatch is visible rather than mysterious.

For review, nbstripout or jupytext help enormously. The raw .ipynb is JSON containing outputs and execution counts, so a diff is unreadable and merge conflicts are common. Stripping outputs before committing, or pairing the notebook with a .py representation, makes the change reviewable.

The structural fix is to keep heavy logic in imported modules and let the notebook be a thin surface over it. Less code in the notebook means less that can be run out of order.

Pickle is not a data format that a reader parses. It is a small instruction stream that the unpickler executes, and part of that instruction set can import a module and call it. A crafted pickle can run an arbitrary command as your user, and it happens during pickle.load, before you have inspected anything.

import pickle
obj = pickle.load(open("model_from_slack.pkl", "rb"))
# code has already run by this point

There is no safe mode, no sandbox flag, and no validation you can do first. The only mitigation is not loading pickles you did not produce, or loading them somewhere disposable.

The second problem bites even with your own files. A pickle stores a reference to the class by module path, not the class definition itself. Load it into an environment where that module has moved, the class has gained a field, or the library has been upgraded, and you get an AttributeError, a ModuleNotFoundError, or worse, an object that reconstructs with the wrong attributes and behaves oddly rather than failing.

That makes pickle poor for anything meant to last. A model pickled with one scikit-learn version and loaded under another is explicitly not supported, and the failure can be silent.

What to use instead depends on the payload. For data, use a format with a real parser: JSON, Parquet, or CSV. For configuration, TOML or YAML. For numeric arrays, numpy.save. For a trained model that must survive an upgrade, an interchange format such as ONNX, or code that rebuilds the model from saved parameters.

Where pickle remains reasonable: short-lived caching inside one process or one pipeline run, in an environment you control, where the file is regenerated rather than archived. Record the library versions with it, because you will need them.

# loads 30 GB into RAM, then dies
rows = open("exports/claims.csv").readlines()

# streams, one line at a time
with open("exports/claims.csv") as f:
    header = next(f)
    for line in f:
        process(line)

A file object is its own iterator, so the for loop reads a buffered chunk from disk and hands you one line per iteration. Memory stays flat regardless of file size. That next(f) for the header works because the same iterator is consumed by both.

For CSV specifically, csv.reader wraps the same streaming behaviour and handles quoted fields containing newlines, which naive line splitting gets wrong. Reach for it rather than line.split(",") on anything that came out of a spreadsheet.

Binary or fixed-size reading needs an explicit loop:

with open(path, "rb") as f:
    while chunk := f.read(1024 * 1024):
        digest.update(chunk)

The walrus operator needs Python 3.8 or later; before that it is a while True with a break.

Two things that decide whether streaming actually works for your problem.

Aggregates are fine if they are incremental. Running totals, counts, min and max all work in one pass with constant memory. A median or an exact distinct count needs the whole dataset, so either accept the memory, spill to a temporary file, or use an approximate algorithm.

The failure mode people hit anyway is accumulating inside the loop. Streaming the file and appending every parsed row to a list uses the same memory as reading it all at once, just more slowly. Write results out as you go, or aggregate into something bounded.

If the work per line is genuinely CPU-heavy, multiprocessing over line ranges helps; if it is I/O bound, it mostly will not, since one disk is the constraint.

class Route:
    def __init__(self, code, stops):
        self.code = code
        self.stops = stops

    def __repr__(self):
        return f"Route({self.code!r}, stops={self.stops})"

    def __str__(self):
        return f"{self.code} ({self.stops} stops)"

r = Route("KA-17", 12)
print(r)         # KA-17 (12 stops)
print([r])       # [Route('KA-17', stops=12)]
r                # Route('KA-17', stops=12)  in the REPL

The list case is the one people get wrong. Containers always call __repr__ on their elements, so a class with only __str__ defined prints beautifully on its own and shows <__main__.Route object at 0x7f...> inside a list, in a debugger, and in a log line that formats a collection. If you write only one of the two, write __repr__.

The !r in the f-string calls repr() on the field, which keeps quotes around strings so the output stays unambiguous.

__eq__ changes comparison from identity to whatever you define:

    def __eq__(self, other):
        if not isinstance(other, Route):
            return NotImplemented
        return self.code == other.code

Returning NotImplemented for unknown types lets Python try the reflected operation instead of forcing a wrong answer.

The consequence people forget: defining __eq__ sets __hash__ to None, so instances become unhashable and cannot go in a set or be used as dict keys. If you need both, define __hash__ over the same fields, and only when those fields are immutable. A hashable object whose hash changes after insertion becomes unfindable in the dict it is sitting in.

@dataclass generates __init__, __repr__ and __eq__ from annotations, and frozen=True adds a __hash__. For plain data-carrying classes it is usually the better starting point.

# ingest.py
def load_readings(path):
    ...

def main():
    load_readings("data/august.csv")
    publish_summary()

if __name__ == "__main__":
    main()

Run python ingest.py and __name__ is "__main__", so main() fires. Write from ingest import load_readings in a notebook or another module, and __name__ is "ingest", so the guard is false and only the definitions are created.

Without it, every import runs the entire job. Someone imports one helper function and the full August ingestion kicks off, writing to the production summary table. It is not a subtle failure, and it is one people meet exactly once.

Testing is the everyday version of the same problem. A test module importing your script to test one function triggers the whole pipeline before any test runs, so the suite is slow and has side effects nobody asked for.

Three related points.

Anything at module level runs on import, guard or not. A database connection or an expensive model load written at the top of the file executes for every importer, so those belong inside main() or a function that is called deliberately.

multiprocessing on Windows and on macOS with the spawn start method re-imports the main module in each child process. Without the guard, each child runs your script again, which spawns more children, and the process count grows until something gives.

Keep main() thin and the guard thinner. Argument parsing goes inside main() rather than at module level, so importing the module does not attempt to read sys.argv from whatever process happens to be running.

tracemalloc is in the standard library and needs no install:

import tracemalloc

tracemalloc.start()
features = build_features(records)
snapshot = tracemalloc.take_snapshot()

for stat in snapshot.statistics("lineno")[:10]:
    print(stat)

current, peak = tracemalloc.get_traced_memory()
print(f"peak {peak / 1e6:.0f} MB")

The statistics name the file and line where each block of memory was allocated, sorted by size. That usually ends the investigation in one run, because a script that dies at 12 GB has one or two lines responsible.

get_traced_memory returning the peak matters more than the current figure. A script can free everything and still have been killed at a moment when three copies of the data existed at once, and a snapshot taken afterwards shows nothing wrong.

For a per-line view of one function, memory_profiler with its @profile decorator prints the increment each statement caused, run under python -m memory_profiler script.py. It is slower than tracemalloc and easier to read for a single hot function.

Two traps worth knowing.

sys.getsizeof measures only the container. A list of one lakh strings reports about 800 KB because it counts the pointers, not the strings they point at. It is useful for a single object and misleading for anything nested.

And a process that peaked at 12 GB may keep a high RSS even after freeing the objects, because the allocator does not always return memory to the OS. So tracemalloc can correctly show the objects gone while top still shows a large process.

Once you know the line, the fix is usually structural: stream instead of materialising, process in chunks, delete intermediates you no longer need, or switch a list of dicts to a more compact representation.

from functools import partial

def resize_crop(image, width, height, interpolation):
    ...

thumb = partial(resize_crop, width=128, height=128,
                interpolation="bilinear")

thumb(img)                        # uses the fixed values
thumb(img, interpolation="cubic") # override at call time

The same thing with a lambda works, and three differences decide between them.

Picklability. A lambda cannot be pickled, so it cannot be sent to a ProcessPoolExecutor or a multiprocessing pool. A partial of a module-level function can. This alone settles most cases in parallel preprocessing code:

with ProcessPoolExecutor() as pool:
    pool.map(partial(resize_crop, width=128, height=128,
                     interpolation="bilinear"), images)

Late binding. Building callables in a loop with a lambda captures the variable, so all of them see the final value. partial stores the argument at construction, so each one keeps its own:

scalers = [partial(rescale, factor=f) for f in factors]

Introspection. thumb.func, thumb.args and thumb.keywords expose what was bound, which a lambda hides entirely. That matters when something logs the callable or a debugger shows it, since a lambda reports only <lambda>.

Two limitations. Positional arguments bind left to right, so fixing the third positional argument means using a keyword instead. And a partial of a partial works but reads badly; one level is usually the limit before a named function is clearer.

Where a lambda still wins: a throwaway expression in a key= or filter, where none of the above matters and the lambda is shorter.

The honest answer is that most experiment code needs functions, not classes, and the useful classes are few and specific.

Where a class earns its place is around state with a lifecycle. Something that is configured, then fitted, then applied, and must remember what it learned:

class ScoreCalibrator:
    def __init__(self, method: str = "isotonic"):
        self.method = method
        self._fitted = None

    def fit(self, raw_scores, outcomes):
        self._fitted = _fit_curve(raw_scores, outcomes,
                                  self.method)
        return self

    def apply(self, raw_scores):
        if self._fitted is None:
            raise RuntimeError("call fit() first")
        return _apply_curve(self._fitted, raw_scores)

The class exists because apply needs what fit learned, and passing that around by hand between notebook cells is exactly how a calibration fitted on the test split ends up applied to itself.

That guard clause is the other benefit. A class can enforce its own ordering, where loose cells cannot enforce anything.

Where classes do not help: a chain of transformations. clean(raw) then featurise(clean) then score(features) are pure functions, trivially testable with a small input, and wrapping them in a class with self.data mutated at each step makes them harder to test and impossible to reuse in a different order.

A class holding only data with no behaviour is a dataclass, and one with a single method is a function with extra syntax.

The refactor to propose in an interview is the hybrid. Move the logic into modules, keep the notebook as a thin surface that imports and calls it:

from experiments.calibration import ScoreCalibrator

The logic becomes testable and version-controllable, the notebook keeps the plots next to the code, and the reproducibility problem of out-of-order cells shrinks to whatever is left in the notebook.

from abc import ABC, abstractmethod

class Forecaster(ABC):
    @abstractmethod
    def fit(self, history): ...

    @abstractmethod
    def predict(self, horizon): ...

    def describe(self):            # concrete, inherited
        return type(self).__name__

class NaiveSeasonal(Forecaster):
    def fit(self, history):
        self.last_cycle = history[-52:]
        return self
    # predict not implemented

NaiveSeasonal()
# TypeError: Can't instantiate abstract class NaiveSeasonal
# with abstract method predict

The failure happens when someone tries to create an instance, not when the module is imported and not when the class is defined. So an incomplete subclass sitting unused in the codebase raises nothing until it is used, which is worth knowing when you expect the error earlier.

An ABC mixes required and provided behaviour freely. describe above is ordinary and every subclass gets it, which is the difference between an ABC and a bare protocol.

Two limits worth stating in an interview.

It checks names, not signatures. A subclass defining predict(self) with no horizon argument satisfies the ABC completely and fails at call time. A type checker catches that; the ABC does not.

And Python does not need an ABC for polymorphism. Duck typing means any object with fit and predict works wherever one is expected, which is exactly how scikit-learn estimators interoperate without inheriting from anything. The ABC buys you an early, loud failure and a single place documenting the contract, at the cost of forcing inheritance.

typing.Protocol, from Python 3.8, is the middle ground: a static checker verifies structural conformance with no inheritance required, and nothing is enforced at runtime.

isinstance still works normally against an ABC, which is the usual reason to keep one when a factory has to dispatch on type.

sum([0.1] * 10) == 1.0        # False
sum([0.1] * 10)               # 0.9999999999999999

Nothing went wrong. 0.1 has no exact binary representation, so each addition carries a rounding error and ten of them accumulate into a visible difference.

import math
math.isclose(sum([0.1] * 10), 1.0)    # True

math.isclose uses a relative tolerance of 1e-09 by default, meaning the values must agree to about nine significant figures. Relative is the right default because the acceptable gap scales with magnitude: a difference of 0.001 is nothing between two values around a crore and enormous between two values around 0.002.

The exception is comparing against zero. Relative tolerance is meaningless there, since any tolerance times zero is zero, so isclose(1e-15, 0.0) returns False. Supply an absolute tolerance:

math.isclose(residual, 0.0, abs_tol=1e-12)

That is the case people get wrong most often, and it turns up whenever you check whether a gradient or a residual has converged.

Two related points for a data science context.

Addition is not associative in floating point, so summing a large array in a different order gives a slightly different total. Parallel reductions and chunked sums therefore produce results that differ in the last digits between runs, which is not a bug and does surprise people comparing outputs across machines.

And accumulated error grows with the number of operations, so a running total over a crore values drifts more than one over a hundred. Compensated summation algorithms exist for when that matters.

For money, floats are the wrong type entirely and Decimal is the answer, but that is a different problem from tolerance comparison.

Python’s integers have no fixed width, so 2 ** 5000 is exact. Floats do not share that property:

import math

1.8e308 * 10          # inf
math.exp(800)         # OverflowError
1e308 * 10 - 1e308 * 10   # nan
float("inf") - float("inf")   # nan
0.0 / 0.0             # ZeroDivisionError in Python

Note the inconsistency: dividing floats by zero raises in pure Python, while the same operation inside numpy produces inf with a warning. That difference alone explains why the same formula behaves differently depending on whether the input is a list or an array.

The propagation is what makes this dangerous. Once one NaN exists, every mean, sum and comparison touching it produces NaN, and nothing raises. A metric reported as NaN at the end of a two-hour training run gives you no indication of which step produced it.

Two behaviours worth knowing precisely.

nan == nan is False, by IEEE definition, since NaN represents an undefined result and two undefined results are not comparable. So you cannot test for it with equality; use math.isnan(x).

And NaN in a sort corrupts the ordering, because every comparison against it returns False, so the sorted output is not sorted in any meaningful way.

The usual sources in modelling code: the log of zero or a negative, exp of a large positive, division where the denominator can be zero, and subtracting two large near-equal quantities. The last one destroys precision even when it does not overflow.

The practical response is to check at boundaries rather than hunt afterwards. Assert that intermediate arrays are finite after each stage, which converts a silent NaN into a loud failure at the step that caused it. Working in log space, and using stabilised formulations such as log-sum-exp, avoids the overflow in the first place.

The seeded approach is reproducible only against unchanged data:

rng = np.random.default_rng(20260819)
sample = rng.choice(applications, size=5000, replace=False)

Add ten thousand applications and rerun with the same seed, and you get a different sample, because the population changed. That is correct behaviour and usually not what you wanted.

Hashing a stable key gives you both properties:

import hashlib

def in_sample(key: str, pct: int, salt: str = "audit-2026") -> bool:
    digest = hashlib.md5(f"{salt}:{key}".encode()).hexdigest()
    return int(digest[:8], 16) % 100 < pct

sample = [a for a in applications if in_sample(a["app_id"], 5)]

Every application’s membership depends only on its own ID, so the sample is reproducible with no seed to record and stable as the population grows. Change the salt and you get an independent sample of the same size, which is how you draw a second non-overlapping cohort later.

Two things to be deliberate about.

Do not use Python’s built-in hash() for this. It is randomised per process for strings, so the same key gives different results in different runs. hashlib is deterministic across processes and versions.

And sampling rows is not sampling entities. Selecting 5% of loan applications over-represents applicants with many applications, so any per-applicant statistic from that sample is biased. Hash the applicant ID instead, and take all their applications.

For stratified sampling, apply the hash within each stratum and take the same percentage from each, which preserves the population mix. Equal-size strata need a different analysis afterwards, with weights.

Whatever you choose, persist the sampled identifiers alongside the analysis. Six months later, “5% hashed on app_id with salt audit-2026” is a reproducible description, and the stored list is the proof.

The test is whether a caller would want to catch it separately. If the answer is no, ValueError with a clear message is enough and adding a class is noise.

class FeedError(Exception):
    """Base for everything this module raises."""

class SchemaMismatch(FeedError):
    def __init__(self, expected, found):
        self.expected = expected
        self.found = found
        super().__init__(
            f"expected {expected} columns, found {found}"
        )

class FeedUnavailable(FeedError):
    pass

The base class is the part that earns its keep. A caller who does not care which specific thing went wrong writes except FeedError: once, and keeps working when you add a fourth subclass next month. Without a shared base, every caller maintains a growing tuple of your exception names.

Carrying structured attributes matters more than the message. A retry layer needs to know whether the failure is transient, and parsing that out of a string is fragile:

except SchemaMismatch as exc:
    if exc.found > exc.expected:
        drop_extra_columns()

Two things worth mentioning in an interview. Inherit from a built-in when the semantics genuinely match and callers may already be catching it: a custom lookup failure inheriting from KeyError keeps existing code working. Do not inherit from BaseException; that is reserved for things that should escape ordinary handlers.

And when wrapping a lower-level error, preserve the chain:

except ConnectionError as exc:
    raise FeedUnavailable("vendor feed down") from exc

from exc attaches the original traceback. Without it, the log shows your exception and nothing about the socket error that caused it, which is exactly the detail you need at 2 AM.

import joblib, sklearn, sys

artefact = {
    "model": fitted_pipeline,
    "sklearn_version": sklearn.__version__,
    "python_version": sys.version.split()[0],
    "trained_on": "2026-08-19",
    "feature_order": feature_names,
    "metrics": {"auc": 0.874, "n_train": 412_000},
}
joblib.dump(artefact, "churn_v7.joblib", compress=3)

Saving a dict rather than the bare estimator is the point. The model alone tells the serving team nothing about what it expects or whether their environment matches yours.

joblib is preferred over plain pickle here because it stores large numeric arrays more efficiently, particularly for estimators holding big coefficient matrices. It is a wrapper over pickle, not a replacement, so unpickling still executes code and version coupling still applies. Never load a joblib file from an untrusted source.

Serialise the whole pipeline, not just the final estimator. If the scaler and encoder were fitted on training data, their fitted state must travel with the model, or serving refits them on different data and the features no longer match what the model learned. That is the most common way a model that validated well performs poorly in production.

feature_order matters more than it looks. Most estimators take a positional array, so a serving path that assembles features in a different order produces confident nonsense with no error at all.

Two operational habits. A version mismatch warning on load is not noise; scikit-learn says explicitly that cross-version loading is unsupported, and the object may work while behaving differently. Match the version or retrain.

And ship a handful of input rows with their expected predictions. The serving team runs them after loading, and a mismatch tells them immediately that something in the environment or the feature assembly is wrong, before any traffic reaches it.

The distinction to have ready: a single % is a line magic applying to the rest of that line, and %% is a cell magic applying to the entire cell and must be the first line of it.

%timeit compute_ndcg(scores, labels)      # repeats, reports spread
%%time                                     # one run, wall and CPU
build_index(corpus)

%timeit loops and reports the best of several runs, which is the honest measurement. %time runs once and is what you want when the operation is slow enough that repeating it is impractical.

For finding where the time goes inside a call:

%prun -l 15 train_ranker(features, labels)

That gives the cProfile output inline, sorted by cumulative time, without leaving the notebook. %lprun from line_profiler goes further and reports per line, after %load_ext line_profiler.

%debug is the one people underuse. Run it in a fresh cell straight after a traceback and it drops you into a post-mortem debugger at the frame where the exception occurred, with all the locals intact. You can inspect the variable that was None without rerunning a twenty-minute cell.

%autoreload matters once your logic lives in imported modules:

%load_ext autoreload
%autoreload 2

Edits to imported files take effect on the next call with no kernel restart. It has real limits: it does not reliably pick up new class attributes, changes to decorators, or module-level constants already bound, so when behaviour looks stale, restart before debugging further.

Two others worth naming. %who and %whos list the variables currently in memory, which is a quick way to see the hidden state accumulated in a long session. And ! runs a shell command, so !pip list | grep scikit checks the environment without leaving the notebook.