Clear, Concise, and Effective Programming
Luciano Ramalho
简介
Don't waste time bending Python to fit patterns you've learned in other languages. Python's simplicity lets you become productive quickly, but often this means you aren't using everything the language has to offer. With the updated edition of this hands-on guide, you'll learn how to write effective, modern Python 3 code by leveraging its best ideas. Discover and apply idiomatic Python 3 features beyond your past experience. Author Luciano Ramalho guides you through Python's core language features and libraries and teaches you how to make your code shorter, faster, and more readable. Complete with major updates throughout, this new edition features five parts that work as five short books within the book: Data structures: Sequences, dicts, sets, Unicode, and data classes Functions as objects: First-class functions, related design patterns, and type hints in function declarations Object-oriented idioms: Composition, inheritance, mixins, interfaces, operator overloading, protocols, and more static types Control flow: Context managers, generators, coroutines, async/await, and thread/process pools Metaprogramming: Properties, attribute descriptors, class decorators, and new class metaprogramming hooks that replace or simplify metaclasses
contents
Preface
Who This Book Is For
Who This Book Is Not For
How This Book Is Organized
Hands-On Approach
Hardware Used for Timings
Soapbox: My Personal Perspective
Python Jargon
Python Version Covered
Conventions Used in This Book
Using Code Examples
How to Contact Us
Acknowledgments
I. Prologue
1. The Python Data Model
What’s new in this chapter
A Pythonic Card Deck
How Special Methods Are Used
Emulating Numeric Types
String Representation
Boolean Value of a Custom Type
Collection API
Overview of Special Methods
Why len Is Not a Method
Chapter Summary
Further Reading
II. Data Structures
2. An Array of Sequences
What’s new in this chapter
Overview of Built-In Sequences
List Comprehensions and Generator Expressions
List Comprehensions and Readability
Listcomps Versus map and filter
Cartesian Products
Generator Expressions
Tuples Are Not Just Immutable Lists
Tuples as Records
Tuples as Immutable Lists
Comparing Tuple and List Methods
Unpacking sequences and iterables
Using * to grab excess items
Nested Tuple Unpacking
Unpacking with * in function calls and sequence literals
Sequence Patterns in match/case
Slicing
Why Slices and Range Exclude the Last Item
Slice Objects
Multidimensional Slicing and Ellipsis
Assigning to Slices
Using + and * with Sequences
Building Lists of Lists
Augmented Assignment with Sequences
A += Assignment Puzzler
list.sort versus the sorted Built-In
When a List Is Not the Answer
Arrays
Memory Views
NumPy
Deques and Other Queues
Chapter Summary
Further Reading
3. Dictionaries and Sets
What’s new in this chapter
Modern dict Syntax
dict Unpacking
dict Patterns in match/case
dict Comprehensions
Standard API of Mapping Types
Overview of Common Mapping Methods
Inserting or updating mutable values
Mappings with Flexible Key Lookup
defaultdict: Another Take on Missing Keys
The __missing__ Method
Variations of dict
Building custom mappings
Subclassing UserDict
Immutable Mappings
Dictionary views
Set Theory
Set Literals
Set Comprehensions
Set Operations
Set operations on dict views
Internals of sets and dicts
A Performance Experiment
Set hash tables under the hood
The hash table algorithm
Hash table usage in dict
Key-sharing dictionary
Practical Consequences of How dict Works
Chapter Summary
Further Reading
4. Text Versus Bytes
What’s new in this chapter
Character Issues
Byte Essentials
Basic Encoders/Decoders
Understanding Encode/Decode Problems
Coping with UnicodeEncodeError
Coping with UnicodeDecodeError
SyntaxError When Loading Modules with Unexpected Encoding
How to Discover the Encoding of a Byte Sequence
BOM: A Useful Gremlin
Handling Text Files
Beware of Encoding Defaults
Normalizing Unicode for Reliable Comparisons
Case Folding
Utility Functions for Normalized Text Matching
Extreme “Normalization”: Taking Out Diacritics
Sorting Unicode Text
Sorting with the Unicode Collation Algorithm
The Unicode Database
Finding characters by name
Numeric meaning of characters
Dual-Mode str and bytes APIs
str Versus bytes in Regular Expressions
str Versus bytes in os Functions
Multi-character emojis
Country flags
Skin tones
Rainbow flag and other ZWJ sequences
Chapter Summary
Further Reading
5. Record-like data structures
What’s new in this chapter
Overview of data class builders
Main features
Classic Named Tuples
Typed Named Tuples
Type hints 101
No runtime effect
Variable annotation Syntax
The meaning of variable annotations
More about @dataclass
Field options
Post-init processing
Typed class attributes
Initialization variables that are not fields
@dataclass Example: Dublin Core Resource Record
Data class as a code smell
Data class as scaffolding
Data class as intermediate representation
Parsing binary records with struct
Structs and Memory Views
Should we use struct?
Chapter Summary
Further Reading
6. Object References, Mutability, and Recycling
What’s new in this chapter
Variables Are Not Boxes
Identity, Equality, and Aliases
Choosing Between == and is
The Relative Immutability of Tuples
Copies Are Shallow by Default
Deep and Shallow Copies of Arbitrary Objects
Function Parameters as References
Mutable Types as Parameter Defaults: Bad Idea
Defensive Programming with Mutable Parameters
del and Garbage Collection
Weak References
The WeakValueDictionary Skit
Limitations of Weak References
Tricks Python Plays with Immutables
Chapter Summary
Further Reading
III. Functions as Objects
7. First-Class Functions
What’s new in this chapter
Treating a Function Like an Object
Higher-Order Functions
Modern Replacements for map, filter, and reduce
Anonymous Functions
The Nine Flavors of Callable Objects
User-Defined Callable Types
Function Introspection
From Positional to Keyword-Only Parameters
Positional-only parameters
Retrieving Information About Parameters
Packages for Functional Programming
The operator Module
Freezing Arguments with functools.partial
Chapter Summary
Further Reading
8. Type Hints in Functions
What’s new in this chapter
About gradual typing
Gradual typing in practice
Starting with Mypy
Making Mypy More Strict
A Default Parameter Value
Using None as a default
Type hints for Python 2.7 and 3.x
Types are defined by supported operations
Types usable in annotations
The Any type
Simple types and classes
Optional and Union types
Generic collections
Tuple
Generic mappings
Abstract Base Classes
Iterable
Parameterized generics and TypeVar
Static Protocols
Callable
NoReturn
Annotating positional-only and variadic parameters
Chapter summary
Further Reading
9. Decorators and Closures
What’s new in this chapter
Decorators 101
When Python Executes Decorators
Registration decorators
Variable Scope Rules
Closures
The nonlocal Declaration
Implementing a Simple Decorator
How It Works
Decorators in the Standard Library
Memoization with functools.cache
Using lru_cache
Single Dispatch Generic Functions
Parameterized Decorators
A Parameterized Registration Decorator
The Parameterized Clock Decorator
A class-based clock decorator
Chapter Summary
Further Reading
10. Design Patterns with First-Class Functions
What’s new in this chapter
Case Study: Refactoring Strategy
Classic Strategy
Function-Oriented Strategy
Choosing the Best Strategy: Simple Approach
Finding Strategies in a Module
Decorator-Enhanced Strategy Pattern
Command
Chapter Summary
Further Reading
IV. Classes and Protocols
11. A Pythonic Object
What’s new in this chapter
Object Representations
Vector Class Redux
An Alternative Constructor
classmethod Versus staticmethod
Formatted Displays
A Hashable Vector2d
Private and “Protected” Attributes in Python
Saving Memory with __slots__
The Problems with __slots__
Overriding Class Attributes
Chapter Summary
Further Reading
12. Sequence Hacking, Hashing, and Slicing
What’s new in this chapter
Vector: A User-Defined Sequence Type
Vector Take #1: Vector2d Compatible
Protocols and Duck Typing
Vector Take #2: A Sliceable Sequence
How Slicing Works
A Slice-Aware __getitem__
Vector Take #3: Dynamic Attribute Access
Vector Take #4: Hashing and a Faster ==
Vector Take #5: Formatting
Chapter Summary
Further Reading
13. Interfaces, Protocols, and ABCs
The Typing Map
What’s new in this chapter
Two kinds of protocols
Programming ducks
Python Digs Sequences
Monkey-Patching: Implementing a Protocol at Runtime
Defensive programming and “fail fast”
Goose typing
Subclassing an ABC
ABCs in the Standard Library
Defining and Using an ABC
ABC Syntax Details
Subclassing the Tombola ABC
A Virtual Subclass of Tombola
Usage of register in Practice
Structural typing with ABCs
Static protocols
The typed double function
Runtime checkable static protocols
Supporting a static protocol
Designing a static protocol
Extending a protocol
Protocol naming conventions
The numbers ABCs and numeric protocols
Chapter Summary
Further Reading
14. Inheritance: For Good or For Worse
What’s new in this chapter
Subclassing Built-In Types Is Tricky
Multiple Inheritance and Method Resolution Order
Multiple Inheritance in the Real World
Coping with Multiple Inheritance
1. Distinguish Interface Inheritance from Implementation Inheritance
2. Make Interfaces Explicit with ABCs
3. Use Mixins for Code Reuse
4. Make Mixins Explicit by Naming
5. An ABC May Also Be a Mixin; The Reverse Is Not True
6. Don’t Subclass from More Than One Concrete Class
7. Provide Aggregate Classes to Users
8. “Favor Object Composition Over Class Inheritance.”
Tkinter: The Good, the Bad, and the Ugly
A Modern Example: Mixins in Django Generic Views
Chapter Summary
Further Reading
15. More About Type Hints
What’s new in this chapter
Overloaded signatures
Max Overload
Takeaways from Overloading max
TypedDict
Type Casting
Reading Type Hints at Runtime
Problems with Annotations at Runtime
Dealing with the Problem
Implementing a generic class
Basic Jargon for Generic Types
Variance
An Invariant Dispenser
A Covariant Dispenser
A Contravariant Trash Can
Variance Review
Implementing a generic static protocol
Chapter summary
Further Reading
16. Operator Overloading: Doing It Right
What’s new in this chapter
Operator Overloading 101
Unary Operators
Overloading + for Vector Addition
Overloading * for Scalar Multiplication
Using @ as an infix operator
Wrapping-up arithmetic operators
Rich Comparison Operators
Augmented Assignment Operators
Chapter Summary
Further Reading
17. Iterables, Iterators, and Generators
What’s new in this chapter
A Sequence of Words
Why Sequences Are Iterable: The iter Function
Iterables Versus Iterators
Sentence classes with __iter__
Sentence Take #2: A Classic Iterator
Don’t make the iterable an iterator for itself
Sentence Take #3: A Generator Function
How a Generator Works
Lazy sentences
Sentence Take #4: Lazy Generator
Sentence Take #5: Lazy Generator Expression
Generator Expressions: When to Use Them
Another Example: Arithmetic Progression Generator
Arithmetic Progression with itertools
Generator Functions in the Standard Library
Subgenerators with yield from
Reinventing chain.
Traversing a tree
Iterable Reducing Functions
A Closer Look at the iter Function
Case Study: Generators in a Database Conversion Utility
Generators as Coroutines
Parameterized Iterable Types
Chapter Summary
Further Reading
18. Context Managers and else Blocks
What’s new in this chapter
Do This, Then That: else Blocks Beyond if
Context Managers and with Blocks
The contextlib Utilities
Using @contextmanager
Chapter Summary
Further Reading
19. Classic Coroutines
What’s new in this chapter
How Coroutines Evolved from Generators
Basic Behavior of a Generator Used as a Coroutine
Example: Coroutine to Compute a Running Average
Decorators for Coroutine Priming
Coroutine Termination and Exception Handling
Returning a Value from a Coroutine
Using yield from
Pipelines of coroutines
The Meaning of yield from
Basic behavior of yield from
Exception handling in yield from
Use Case: Coroutines for Discrete Event Simulation
About Discrete Event Simulations
The Taxi Fleet Simulation
Generic Type Hints for Classic Coroutines
Chapter Summary
Further Reading
20. Concurrency Models in Python
What’s new in this chapter
A Bit of Jargon
Processes, threads, and Python’s Infamous GIL
A Concurrent Hello World
Spinner with threading
Spinner with multiprocessing
Spinner with asyncio
Supervisors Side-by-side
The Real Impact of the GIL
Quick Quiz
A Homegrown Process Pool
Process-based Solution
Understanding the Elapsed Times
Code for the Multi-core Prime Checker
Thread-based Non-solution
The Big Picture
System Administration
Data Science
Server-side Web/Mobile Development
WSGI Application servers
Distributed task queues
Chapter Summary
Further Reading
Concurrency with threads and processes
The GIL
Concurrency beyond the standard library
Concurrency and scalability beyond Python
21. Concurrency with Futures
What’s new in this chapter
Concurrent Web Downloads
A Sequential Download Script
Downloading with concurrent.futures
Where Are the Futures?
Launching Processes with concurrent.futures
Multi-core Prime Checker Redux
Experimenting with Executor.map
Downloads with Progress Display and Error Handling
Error Handling in the flags2 Examples
Using futures.as_completed
Chapter Summary
Further Reading
22. Asynchronous Programming
What’s New in this Chapter
A few definitions
Example: Probing Domains
Guido’s trick to read asynchronous code
New concept: awaitable
Downloading with asyncio and aiohttp
The Secret of Native Coroutines: Humble Generators
The all-or-nothing problem
Asynchronous Context Managers
Enhancing the asyncio downloader
Using asyncio.as_completed and a semaphore
Using an Executor to Avoid Blocking the Event Loop
Making Multiple Requests for Each Download
Writing asyncio Servers
A FastAPI Web Service
An asyncio TCP Server
Asynchronous iteration and asynchronous iterables
Asynchronous Generator Functions
Async Comprehensions and Async Generator Expressions
Generic Asynchronous Types
Async beyond asyncio: Curio
How Async Works and How It Doesn’t
Running Circles Around Blocking Calls
The Myth of I/O Bound Systems
Avoiding CPU-bound Traps
Chapter Summary
Further Reading
23. Dynamic Attributes and Properties
What’s new in this chapter
Data Wrangling with Dynamic Attributes
Exploring JSON-Like Data with Dynamic Attributes
The Invalid Attribute Name Problem
Flexible Object Creation with __new__
Computed Properties
Step 1: Data-driven Attribute Creation
Step 2: Property to Retrieve a Linked Record
Step 3: Property Overriding an Existing Attribute
Step 4: Bespoke Property Cache
Step 5: Caching Properties with functools
Using a Property for Attribute Validation
LineItem Take #1: Class for an Item in an Order
LineItem Take #2: A Validating Property
A Proper Look at Properties
Properties Override Instance Attributes
Property Documentation
Coding a Property Factory
Handling Attribute Deletion
Essential Attributes and Functions for Attribute Handling
Special Attributes that Affect Attribute Handling
Built-In Functions for Attribute Handling
Special Methods for Attribute Handling
Chapter Summary
Further Reading
24. Attribute Descriptors
What’s new in this chapter
Descriptor Example: Attribute Validation
LineItem Take #3: A Simple Descriptor
LineItem Take #4: Automatic Storage Attribute Names
LineItem Take #5: A New Descriptor Type
Overriding Versus Non-Overriding Descriptors
Overriding Descriptors
Overriding Descriptor Without __get__
Non-overriding Descriptor
Overwriting a Descriptor in the Class
Methods Are Descriptors
Descriptor Usage Tips
Descriptor docstring and Overriding Deletion
Chapter Summary
Further Reading
