Oregon State University
2025-01-22
This does:
data.txt existspos to start of the file, pos = 0This does not read any of the file into memory, write anything to the file, or close the file.
f.read(n=-1): Reads in n bytes; if n=-1 or not present, read entire rest of file.f.readline(): Read next full line, return string with newline character.f.readlines(): Reads entire rest of file, returns list of strings (with newlines).f.seek(pos): Move file cursor to specified position.f.tell(): Return current position in file.f.write(s): Insert string s at current position.f.flush(): Perform all pending write operations.f.close(): Close file (no more reading or writing).matrix.txt contents:
f = open('matrix.txt')
matrix = []
for line in f.readlines():
row = [int(x) for x in line.split(',')]
matrix.append(row)
f.close()Make sure to close the file!
Use context manager & with to automatically close:
When opening a file, by default it opens in read-only mode. Other file modes:
'r': Read-only, no writing. Starts at pos = 0.'w': Write. Creates file if it doesn’t exist (if it does, contents are deleted). Starts at pos = 0.'a': Append. Opens file for writing but does not delete contents; creates file if it doesn’t exist. Starting pos is end of file.'+': Update. Opens for reading and writing, does not delete current contents. Starts at pos = 0.Basic idea: better to store structured, numerical data in binary formats over plain-text ASCII files. Why? Smaller.
# small ints # medium ints
42 (4 bytes) 123456 (4 bytes)
'42' (2 bytes) '123456' (6 bytes)
# near-int floats # e-notation floats
12.34 (8 bytes) 42.424242E+42 (8 bytes)
'12.34' (5 bytes) '42.424242E+42' (13 bytes)Also, faster I/O, since binary files save in native format.
HDF5 files (.hdf5, .h5) store data in binary format.
HDF5 provides database features like storing many datasets, user-defined metadata, optimized I/O, and ability to query contents.
HDF5 is a filesystem in a file: provides nested tree structure for datasets.
import tables as tb
PyTables provides five basic dataset classes:
Array: homogeneous components, fixed size; great for numerical dataCArray: chunked arraysEArray: extendable arraysVLArray: variable-length arraysTable: structured collection of records whose values are stored in fixed-length fieldsConstructs need to be composed of atomic types:
bool: true or false typeint: signed integer typesuint: unsigned integer typesfloat: floating-point typescomplex: complex floating-point typesstring: fixed-length raw string typeAlso: Groups, links, and hidden nodes
… or, better yet:
'r': Read-only; no data can be modified.'w': Write; create a new file (delete existing file with that name).'a': Append; open existing file for reading or writing, or create new file.'r+': Similar to a, but file must exist.HDF5 files: all nodes stem from root node: / or h5file.root
Natural naming: can access subnodes as attributes of “parent” nodes, like h5file.root.a_group.some_data
(all relevant nodes in the tree must have names that are valid Python variable names)
Create and access a group on the root node:
Datasets: arrays and tables, each with a corresponding create method:
# integer array
h5file.create_array(h5file.root.a_group, 'arthur_count', [1, 2, 5, 3])
# tables need descriptions
dt = np.dtype([('id', int), ('name', 'S10')])
knights = np.array([(42, 'Lancelot'), (12, 'Bedivere')], dtype=dt)
h5file.create_table(h5file.root, 'knights', dt)
h5file.root.knights.append(knights)h5file.root.a_group.arthur_count[:]
type(h5file.root.a_group.arthur_count[:])
type(h5file.root.a_group.arthur_count)h5file.root.knights[1] # pull out second row
h5file.root.knights[:1] # slice the first row
mask = (h5file.root.knights.cols.id[:] < 28)
h5file.root.knights[mask] # create mask and apply to table
# Fancy index the second and first rows, in that order
h5file.root.knights[([1, 0],)]Memory mapping: pull in data from disk only as needed; HDF5 takes care of this automatically
table_def = {'time': tables.Float64Col(pos=0),
'temperature': tables.Float64Col(pos=1),
'pressure': tables.Float64Col(pos=2),
}
with tables.open_file(filename, mode='w', title="Table title"
) as h5file:
table = h5file.create_table(where=h5file.root,
name='simulation',
description=table_def
)
# sim is an object representing a simulation
timestep = table.row
# Save initial conditions
timestep['time'] = sim.time
timestep['temperature'] = sim.temperature
timestep['pressure'] = sim.pressure
timestep.append()
# Main time integration loop
while sim.time < time_end:
sim.step() # perform a single integration step
# Save new timestep information
timestep['time'] = sim.time
timestep['temperature'] = sim.temperature
timestep['pressure'] = sim.pressure
# Add ``timestep`` to table
timestep.append()
table.flush()with tables.open_file(filename, 'r') as h5file:
# Load Table with Group name simulation
table = h5file.root.simulation
time = table.col('time')
pressure = table.col('pressure')
temperature = table.col('temperature')These are NumPy arrays!
# particles: id, kind, velocity
particles = [(42, 'electron', 72.0),
(43, 'proton', 0.1),
(44, 'electron', 76.8),
(45, 'neutron', 0.39),
(46, 'neutron', 0.72),
(47, 'neutron', 0.55),
(48, 'proton', 0.18),
(49, 'neutron', 0.23),
...
]If we know we want to look at neutral and charged particles separately, then why search through all the particles all the time?
neutral = [(45, 'neutron', 0.39),
(46, 'neutron', 0.72),
(47, 'neutron', 0.55),
(49, 'neutron', 0.23),
...
]
charged = [(42, 'electron', 72.0),
(43, 'proton', 0.1),
(44, 'electron', 76.8),
(48, 'proton', 0.18),
...
]But now kind is redundant in the neutral table. Let’s delete this column and rely on the structure of the tables together to dictate which table refers to what.
neutral = [(45, 0.39),
(46, 0.72),
(47, 0.55),
(49, 0.23),
...
]
charged = [(42, 'electron', 72.0),
(43, 'proton', 0.1),
(44, 'electron', 76.8),
(48, 'proton', 0.18),
...
]We are embedding information directly into the semantics of the hierarchy.
We could add another layer that distinguishes the particles based on detector:
Data should be broken up like this to improve access time speeds. This is more efficient because there are: fewer rows to search, fewer rows to pull from disk, and fewer columns in description (decreases size of rows).
Rather than only calling your package from another Python code, you can (fairly easily)create a command-line interface.
You can and should also use these to change inputs or settings to your programs without modifying code.
# contents of example.py
# import the necessary packages
import sys
import argparse
# construct the argument parse and parse the arguments
parser = argparse.ArgumentParser(
description='This is a simple command-line program.'
)
parser.add_argument('-n', '--name', required=True,
help='name of the user'
)
args = parser.parse_args(sys.argv[1:])
# display a friendly message to the user
print("Hi there {}, it's nice to meet you!".format(args.name))For larger/more complex input, may be better to use an input file.
YAML is a good option for this: use pyyaml (import yaml)
input.yaml:
Returns a dictionary