#220 What, why, and where of friendly errors in Python

Sponsored by Datadog: pythonbytes.fm/datadog

Special guest: Hannah Stepanek

Watch on YouTube

Michael #1: We Downloaded 10,000,000 Jupyter Notebooks From Github – This Is What We Learned

• by Alena Guzharina from JetBrains
• Used the hundreds of thousands of publicly accessible repos on GitHub to learn more about the current state of data science. I think it’s inspired by work showcased here on Talk Python.
• 2 years ago there were 1,230,000 Jupyter Notebooks published on GitHub. By October 2020 this number had grown 8 times, and we were able to download 9,720,000 notebooks. 8x growth.
• Despite the rapid growth in popularity of R and Julia in recent years, Python still remains the most commonly used language for writing code in Jupyter Notebooks by an enormous margin.
• Python 2 went from 53% → 11% in the last two years.
• Interesting graphs about package usage
• Not all notebooks are story telling with code: 50% of notebooks contain fewer than 4 Markdown cells and more than 66 code cells.
• Although there are some outliers, like notebooks with more than 25,000 code lines, 95% of the notebooks contain less than 465 lines of code.

Brian #2: pytest-pythonpath

• plugin for adding to the PYTHONPATH from the pytests.ini file before tests run
• Mentioned briefly in episode 62 as a temporary stopgap until you set up a proper package install for your code. (cringing at my arrogance).
• Lots of projects are NOT packages. For example, applications.
• I’ve been working with more and more people to get started on testing and the first thing that often comes up is “My tests can’t see my code. Please fix.”
• Example
  • proj/src/stuff_you_want_to_test.py
  • proj/tests/test_code.py
  • You can’t import stuff_you_want_to_test.py from the proj/tests directory by default.
• The more I look at the problem, the more I appreciate the simplicity of pytest-pythonpath
• pytest-pythonpath does one thing I really care about:
  • Add this to a pytest.ini file at the proj level:

[pytest] python_paths = src

• That’s it. That’s all you have to do to fix the above problem.
• Paths relative to the directory that pytest.ini is in. Which should be a parent or grandparent of the tests directory.
• I really can’t think of a simpler way for people to get around this problem.

Hannah #3: Thinking in Pandas

• Pandas dependency hierarchy (simplified):
  • Pandas -> NumPy -> BLAS (Basic Linear Algebra Subprograms)
• Languages:
-

- Python -> C -> Assembly df["C"] = df["A"] + df["B"] A = [ 1 4 2 0 ] B = [ 3 2 5 1 ] C = [ 1 + 3 4 + 2 2 + 5 0 + 1 ]

• Pandas tries to get the best performance by running operations in parallel.
• You might think we could speed this problem up by doing something like this:

Thread 1: 1 + 3 Thread 2: 4 + 2 Thread 3: 2 + 5 Thread 4: 0 + 1

• However, the GIL (Global Interpreter Lock) prevents us from achieving the performance improvement we are hoping for.
• Below is an example of a common threading problem and how a lock solves that problem.
-

Thread 1 total Thread 2 1 + 3 + 4 + 2 0 0 + 5 10 0 + 6 + 2 total += 10 0 13 total =10 0 total += 13 10 total = 13 13 Thread 1 total Thread 2 1 + 3 + 4 + 2 0 unlocked 0 + 5 10 0 unlocked + 6 + 2 total += 10 0 locked 13 total =10 0 locked 10 unlocked 10 locked total += 13 10 locked total = 13 23 unlocked

• As it turns out, because Python manages memory for you every object in Python would be subject to these kinds of threading issues:

a = 1 # reference count = 1 b = a # reference count = 2 del(b) # reference count = 1 del(a) # reference count = 0

• So, the GIL was invented to avoid this headache which only lets one thread run at a time.
• Certain parts of the Pandas dependency hierarchy are not subject to the GIL (simplified):
  • Pandas -> NumPy -> BLAS (Basic Linear Algebra Subprograms)
  • GIL -> no GIL -> hardware optimizations
• So we can get around the GIL in C land but what kind of optimizations does BLAS provide us with?
  • Parallel operations inside the CPU via Vector registers
• A vector register is like a regular register but instead of holding one value it can hold multiple values.

| 1 | 4 | 2 | 0 | + + + + | 3 | 2 | 5 | 1 | = = = = | 4 | 6 | 7 | 1 |

• Vector registers are only so large though, so the Dataframe is broken up into chunks and the vector operations are performed on each chunk.

Michael #4: Quickle

• Fast. Benchmarks show it’s among the fastest serialization methods for Python.
• Safe. Unlike pickle, deserializing a user provided message doesn’t allow for arbitrary code execution.
• Flexible. Unlike msgpack or json, Quickle natively supports a wide range of Python builtin types.
• Versioning. Quickle supports “schema evolution”. Messages can be sent between clients with different schemas without error.
• Example

>>> import quickle >>> data = quickle.dumps({"hello": "world"}) >>> quickle.loads(data) {'hello': 'world'}

Brian #5: what(), why(), where(), explain(), more() from friendly-traceback console

• Do this:

$ pip install friendly-friendly_traceback.install() $ python -i >>> import friendly_traceback >>> friendly_traceback.start_console() >>>

• Now, after an exception happens, you can ask questions about it.

>>> pass = 1 Traceback (most recent call last): File "[HTML_REMOVED]", line 1 pass = 1 ^ SyntaxError: invalid syntax >>> what() SyntaxError: invalid syntax A `SyntaxError` occurs when Python cannot understand your code. >>> why() You were trying to assign a value to the Python keyword `pass`. This is not allowed. >>> where() Python could not understand the code in the file '[HTML_REMOVED]' beyond the location indicated by --> and ^. -->1: pass = 1 ^

• Cool for teaching or learning.

Hannah #6: Bandit

• Bandit is a static analysis security tool.
• It’s like a linter but for security issues.

pip install bandit bandit -r .

• I prefer to run it in a git pre-commit hook:

# .pre-commit-config.yaml repos: repo: https://github.com/PyCQA/bandit rev: '1.7.6' hooks: - id: bandit

• It finds issues like:
  • flask_debug_true
  • request_with_no_cert_validation
• You can ignore certain issues just like any other linter:

assert len(foo) == 1 # nosec

Extras:

Brian:

• Meetups this week 2/3 done.
  • NOAA Tuesday, Aberdeen this morning - “pytest Fixtures”
  • PDX West tomorrow - Michael Presenting “Python Memory Deep Dive”
• Updated my training page, testandcode.com/training
  • Feedback welcome.
  • I really like working directly with teams and now that trainings can be virtual, a couple half days is super easy to do.

Michael:

• PEP 634 -- Structural Pattern Matching: Specification accepted in 3.10
• PyCon registration open
• Python Web Conf reg open
• Hour of code - minecraft

Joke:

Sent in via Michel Rogers-Vallée, Dan Bader, and Allan Mcelroy. :)

PEP 8 Song

Watch on YouTube

• By Leon Sandoy and team at Python Discord