The shatterdome#

In the shatterdome we’ll have a closer look at some of the internal mechanics of the jaeger.

The `CAN bus`#

The JaegerCAN class provides the lowest level access to the positioners via the CAN bus. JaegerCAN provides access to the appropriate bus subclass, while also adding including jaeger functionality. Normally JaegerCAN is instantiated when FPS is and you won’t have to use it unless you want to access the bus directly.

JaegerCAN can be instantiated by passing it an interface and the parameters necessary to instantiate the corresponding CAN bus. interface must be one of INTERFACES, which defines the correlation between interfaces and CAN buses. For instance, to create a CAN@net bus we do

>>> can = JaegerCAN('cannet', channel='10.1.25.100', bitrate=1000000)
>>> isinstance(bus.interfaces[0], CANNetBus)
True

Note that some interfaces require having python-can installed. The main interfaces (CANNetBus and VirtualBus) do not require python-can and are in fact reimplemented to optimise how they behave with asyncio.

Loading from a profile#

The configuration file contains a section in which multiple bus interfaces can be defined. An example of bus profile is

profiles:
    default: cannet
    cannet:
        interface: cannet
        channels: [192.168.0.10]
        port: 19228
        buses: [1, 2, 3, 4]
        bitrate: 1000000
    slcan:
        interface: slcan
        channel: /dev/tty.usbserial-LW3HTDSY
        ttyBaudrate: 1000000
        bitrate: 1000000

These configurations can be loaded by using the JaegerCAN.from_profile classmethod

>>> bus = JaegerCAN.from_profile('test')
>>> bus
<jaeger.can.JaegerCAN at 0x117594128>

The default profile can be loaded by calling from_profile without arguments. Note that in the case of cannet we define multiple interfaces as a list of channels instead of as a single channel. We’ll talk about multiple interfaces in Multibus interfaces.

The command queue#

Because we need to be able to associate replies from the bus with the command that triggered them, and given that commands and replies don’t have unique identifiers beyond the command and positioner ids, we do not allow more than one instance the pair (command_id, positioner_id) to run at the same time. When a command is executed (ultimately by calling FPS.send_command), the command is put in a queue. When a new command is available in the queue, the code checks that no other command with the same command_id and positioner_id are already running. If no identical command is running, all the messages from the command are sent to the bus and the command remains in running_commands until it has been completed (see the command-done section for more details). If a command is running, the new command is re-queued until the previous command has finished.

Broadcast commands are a bit special: when a broadcast command (positioner_id=0) is running no other command with the same command_id will run until the broadcast has finished, regardless of positioner_id.

Multibus interfaces#

Some CAN devices provide multiple buses (for example, the Ixxat CAN@net device). In addition, the positioners in the FPS may not form a single CAN network since they can be connected to different buses in different devices. JaegerCAN provides support for multichannel and multibus CAN networks. Because these terms can sometimes be confusing, we assume the following nomenclature:

A CAN device is called an interface, which may consist of one or multiple buses.
Each interface is defined by its channel or route to it. The channel can be a TCP address, a device path, etc. Some interfaces require more parameters to define the connection method (for example, a TCP port). Sometimes we use channel and interface as synonyms.
A bus is the minimal CAN network unit. All positioners connected to the same bus belong to the same CAN network.

In jaeger, the JaegerCAN instance represents the entirety of the CAN network, even when it’s composed of multiple interfaces with several buses each. The attribute interfaces contains a list of all the loaded interfaces. At this point, jaeger does not support mixing interfaces of different types.

Whether an interface is multibus or not is defined in INTERFACES. The buses to be used can be defined to JaegerCAN via the buses argument. An example of a multibus interface is CANNetBus.

The mapping between positioners and buses is done in the FPS class. When FPS is instantiated using multibus interfaces (or multiple single bus interfaces), a GET_ID command is broadcast to all the available interfaces and buses. The replies from the positioners are used to create a positioner_to_bus mapping. Because we need to know from what interface and bus the messages originate from, it is assumed that a multibus interface appends interface and bus attributes to the returned messages.

The `FPS` class#

The FPS class is the main entry point to monitor and command the focal plane system and usually it will be the first thing you instantiate. It contains a CAN bus, a dictionary of all the positioners that have been added to the FPS with their associated positioner_id and central position; it can be stored as a file or in a database), and high level methods to perform operations that affect multiple positioners (e.g., send a trajectory).

To instantiate with the default options, simply do

>>> from jaeger import FPS
>>> fps = FPS()

This will create a new CAN bus (accessible as FPS.can) and an empty dictionary of positioners. Note that the CAN bus is not initialised until FPS.start_can or FPS.initialise are called.

Initialisation#

Once we have created a FPS object we’ll need to initialise it by calling and awaiting FPS.initialise. This will issue two broadcast commands: GetStatus and GetFirmwareVersion. The replies to these commands are used to determine which positioners are connected, add them to the FPS, and set their status. Each one of the positioners that have replied are subsequently initialised as detailed in Positioner initialisation.

Sending commands#

The preferred way to send a command to the bus is by using the FPS.send_command method which accepts a commands.CommandID (either as a flag, integer, or string), the list of positioner_ids that we want to address, and additional arguments to be passed to the command associated with the CommandID. For example, to broadcast a GET_ID command

>>> await fps.send_command('GET_ID', positioner_ids=0)

Note that you need to await the command, which will return the execution to the event loop until the command has finished.

Some commands, such as SetActualPosition take multiple attributes

>>> cmd = await fps.send_command(CommandID.SET_ACTUAL_POSITION, positioner_ids=[4, 5], alpha=0, beta=0)
>>> cmd
<Command SET_ACTUAL_POSITION (positioner_ids=[4, 5], status='DONE')>

When a command is send FPS puts it in the bus command queue and, once it gets processed, starts listening for replies from the bus. When it gets a reply with the same command_id and positioner_id the bus sends it to the command for further processing.

Shutting down the FPS#

Positioner pollers and queue watchers are built as Tasks that run forever. If you are executing your code with asyncio.run or run_until_complete, your funcion will never finish and you’ll need to cancel the execution. To cancel all pending tasks and close the FPS object cleanly, run

await fps.shutdown()

FPS as a context manager#

It’s possible to use the FPS object as an async context manager. The FPS is initialised when entering the context and shut down on exit

fps = FPS()
async with fps:
    await fps[13].goto(10, 10)

Adding a single positioner#

Normally one wants FPS.initialise to autodiscover all the positioners connected to all the CAN interfaces. In some cases one may want a lower level control, adding positioners manually. That can be achieves with the following initialisation

>>> p20 = Positioner(20)
>>> fps = FPS()
>>> await fps.start_can()  # Initialise the CAN interfaces manually.
>>> fps.add_positioner(p20, interface=fps.can.interfaces[3], bus=3)
>>> await p20.initialise()
>>> p20.alpha, p20.beta
0.009910762310028076 180.0494384765625

The interface= and bus= parameters to add_positioner can be skipped if JaegerCAN has a single, non-multibus interface.

Sending trajectories#

Trajectories can be sent either a YAML file or a dictionary. In both cases the trajectory must include, for each positioner, a list of positions and times for the 'alpha' arm in the format \(\rm [(\alpha_1, t_1), (\alpha_2, t_2), ...]\), and a similar dictionary for 'beta'. An example of YAML file with a valid trajectory for positioners 1 and 4 is

1:
    alpha: [[20, 5], [100, 10], [50, 15]]
    beta: [[90, 15], [85, 18]]
4:
    alpha: [[200, 3], [100, 15]]
    beta: [[50, 5]]

And it can be commanded by doing

>>> await fps.send_trajectory('my_trajectory.yaml')

Aborting all trajectories#

Trajectories or go to commands can be cancelled for all positioners by using the FPS.abort method

>>> await fps.send_trajectory('my_trajectory.yaml')
>>> await fps.abort()  # Cancel the trajectory

Note that the abort method creates and returns a Task and will be executed even without it being awaited, as long as there is a running event loop. However, it is safer to await the returned task.

`Positioner`, status, and position#

The Positioner class stores information about a single positioner, its status and position, and provides high level methods to command the positioner. Positioner objects need to be linked to a FPS instance and are usually created when the FPS class is instantiated.

Positioner initialisation#

When a Positioner is instantiated it contains no information about its position (angle of the alpha and beta arms) and its status is set to UNKNOWN. By calling and awaiting Positioner.initialise, the following steps are executed:

Updates the firmware version.
The status is updated by calling Positioner.update_status.
Stops all possible trajectories remaining in the buffer for that positioner.
Sets the alpha and beta arm speeds to the default value (stored in the configuration file as motor_speed).

After this sequence, the positioner is ready to be commanded.

Position and status pollers#

The status of the positioner, given as a maskbit PositionerStatusV4_1 (or maskbits.BootloaderStatus if the positioner is in bootloader mode) can be accessed via the status attribute and updated by calling the update_status coroutine. Similarly, the current position of the positioner is stored in the alpha and beta attributes, in degrees, and updated via update_position.

As we initialise the FPS, two Poller instances are created as part of the PollerList FPS.pollers to track the position and status of each positioner. These tasks simply call update_status. and update_position every few seconds and update the corresponding attributes in the positioners. The delay between polls can be set via the set_delay method.

Sending a positioner to a position#

The Positioner.goto coroutine allows to easily send the positioner to a position or set the speed of either arm

await positioner.goto(alpha=30, beta=90, speed=(1000, 1200))

# Only set speed
await positioner.set_speed(500, 500)

# Only go to position using the speed we just set
await positioner.goto(alpha=100, beta=154)

Awaiting Positioner.goto blocks until the positioner has arrived to the desired position and DISPLACEMENT_COMPLETED is set.

Waiting for a status#

In many cases it’s convenient to asynchronously block the execution of a coroutine while we wait until certain bits appear in the status. To do that one can use wait_for_status

# Wait until DISPLACEMENT_COMPLETED appears
await positioner.wait_for_status(PositionerStatusV4_1.DISPLACEMENT_COMPLETED)

# Wait untils SYSTEM_INITIALIZED and DATUM_ALPHA_INITIALIZED are set. Time-out in 3 seconds if that doesn't happen.
await positioner.wait_for_status([PositionerStatusV4_1.SYSTEM_INITIALIZED, PositionerStatusV4_1.DATUM_ALPHA_INITIALIZED], timeout=3)

Note that wait_for_status is independent of the status poller. While wait_for_status is running, a GET_STATUS command will be issue wach delay seconds, in addition to the normal polling.

Commands#

Command provides a base class to implement wrappers around firmware commands. It handles the creation of messages to be passed to the bus, encodes the arbitration id from the command_id` and ``positioner_id, processes replies, and keeps a record of the status of a command. Commands that accept extra data (e.g., positions of the alpha and beta arms) also do the encoding of the input parameters to the format that the firmware command understands, making them easier to use. Commands are asyncio.Future objects and can be awaited until complete. A list of all the available commands can be found here.

Commands can sent directly to the FPS

>>> from jaeger.commands import GetStatus
>>> status_cmd = GetStatus(positioner_ids=4)
>>> status_cmd
<Command GET_STATUS (positioner_ids=4, status='READY')>
>>> fps.send_command(status_cmd)
True
>>> await status_cmd

This is what happens when you execute the above snippet:

When created, the command has status READY and is prepared to be sent to the bus.
When we send_command the command, it gets put in the bus queue.
Shortly after, the bus processes the command from the queue and checks that no other command with the same (command_id, positioner_id) is running. If that’s the case the command status is changed to RUNNING and all the SuperMessage that compose the command are sent to the bus. A SuperMessage is just a wrapper that contains the arbitration_id and the data to send as bytes. Most command will issue just a message but some such as SendTrajectoryData can send multiple messages.
The bus listens to replies from the bus and redirects them to the command with the matching (command_id, positioner_id) where they are processed.
Once the expected replies have been received, or when the command times out, the command is marked DONE or FAILED. See the When is a command marked done? section for more details.
When the command is marked done, the result of the Future is set and the event loop returns.

Replies#

When a reply is received from the bus it is redirected to appropriate command, processed, and stored in the replies list as a Reply object. Reply instances are quite simple and contain the associated positioner_id and command_id as well as the data returned (as a bytearray), and the response_code (and instance of ResponseCode) for the command sent.

When is a command marked done?#

There are several ways in which a command can be marked done:

If the command is not a broadcast and it has received as many replies as messages sent and all those replies have the COMMAND_ACCEPTED bit, then the command is marked DONE. This happens because we expect each message sent to receive a confirmation that it has been accepted, even if the reply doesn’t include any additional data.
If any reply to the command has a ResponseCode different from COMMAND_ACCEPTED then the command is immediately marked FAILED and all additional replies are ignored.
If the command is a broadcast we don’t know how many replies to expect. In that case the command waits until it times out and it’s marked DONE if it has received at least one reply, otherwise FAILED.
If the command is instantiated with timeout=0, the command is marked done the moment it is processed by the bus queue. In this case all replies to the command are ignored.

Time-outs#

When the command is set to RUNNING (i.e., when it is processed from the bus queue), a timer starts that times out the command after a certain delay (usually one second). The timeout can be set when the command is instantiated. When the command times out it is marked done (if is has not already been so) according to the above logic.

The timeout can be set to None, in which case the command will never time out. When combined with a broadcast this means the command will never be marked finished and the user will need to manually call finish_command to finish it. For example

import asyncio

from jaeger import FPS
from jaeger.maskbits import CommandStatus, PositionerStatusV4_1

async def check_status(status_cmd, positioners):

    print('Starting monitoring')

    if all(asyncio.gather(*[positioner.wait_for_status(PositionerStatusV4_1.DATUM_ALPHA_INITIALIZED) for positioner in positioners])):
        status_cmd.finish_command(status=CommandStatus.DONE)
    else:
        status_cmd.finish_command(status=CommandStatus.FAILED)

async def get_status():

    fps = FPS()
    await fps.initialise()

    status_cmd = fps.send_command('GET_STATUS', positioner_ids=0, timeout=None)

    asyncio.create_task(check_status(status_cmd, fps.positioners))

    await status_cmd

    print('Command done')

asyncio.run(get_status())

Accessing the IEB#

The Instrument Electronics Box (IEB) can be accessed via the FPS.ieb attribute. This field is populated by an IEB instance, which is a very thin wrapper around the sdss-drift package. The configuration for the IEB is defined in a YAML configuration file following the format required by drift and then passed to FPS on instantiation or, more frequently, specified in jaeger’s configuration file in the fps.ieb field.

Once IEB has been loaded it behaves like any other Drift instance, and we refer to the documentation there. As an example, we can switch the status of the SYNC line by doing

>>> sync = fps.ieb.get_device('SYNC')
>>> await sync.read()
('open', False)
>>> await sync.switch()
>>> await sync.read()
('closed', False)

Note that here open and closed refer to the status of the relay that controls the SYNC line.

It’s also possible to access the IEB via the actor command ieb.

Internals#

Configuration files#

jaeger uses the default configuration file system from the SDSS Python template. The main configuration file, in YAML format, is included with the package in etc/jaeger.yml. Any section in this file can be overridden in a personal configuration file that must be located at ~/.jaeger.jaeger.yml in the HOME directory of the user executing the code. For example, if the default interfaces section is

profiles:
    default: slcan
    slcan:
        interface: slcan
        channel: /dev/tty.usbserial-LW1FJ8ZR
        ttyBaudrate: 1000000
        bitrate: 1000000
    test:
        interface: test
        channel: none
        ttyBaudrate: 1000000
        bitrate: 1000000

But we want to change the channel of the default configuration we can create a file that contains

interfaces:
    default:
        channel: /dev/tty.USB0

Logging#

There are two loggers in jaeger. Both of them are output to the terminal (with different logging levels) and stored in files. The first one logs all jaeger specific messages and it is written to ~/.jaeger/jaeger.log; by default messages with logging level equal or greater than WARNING are also output to the console. The second log track interactions with the CAN bus and saves messages to ~/.jaeger/can.log; ERROR messages are also output to the console. The logger instances can be access from the top jaeger module by importing from jaeger import log, can_log.

To change the terminal logging level you can use the setLevel method. For instance

import logging
from jaeger import can_log, log

# log.sh contains the terminal logging handler
log.sh.setLevel(logging.DEBUG)
can_log.sh.setLevel(logging.INFO)

Similarly, the file logger can be accessed as log.fh or can_log.fh. To disable all logging you can do

log.propagate = False
can_log.propagate = False

or for a specific handler

log.sh.propagate = False

Note that although warnings issues with the warnings module are redirected to the logging system, but they may need to be silenced independently by doing something like

import warnings
warnings.simplefilter('ignore')

The bootloader mode#

During the first 10 seconds after a positioner has been powered up it remains in bootloader mode. In this state is is possible to issue several specific commands to update the firmware. In this mode the GetStatus command returns bits that must be interpreted using the BootloaderStatus maskbit.

Is is possible to know whether a positioner is in bootloader mode by getting the firmware version command and getting the version string. If the version is 'XX.80.YY' the positioner is in bootloader mode.

Note

This implementation is temporary and will be changed once the bootloaded mode can be set via de sync cable.

Upgrading firmware#

If is possible to upgrade the firmware of a positioner (or set of them) by using the convenience function load_firmware. A CLI interface to this function is available via the jaeger command, for example

jaeger upgrade-firmware ~/Downloads/tendo_v04.00.04.bin

This will sequentially cycle each one of the IEB sextant power supplies and upgrade the firmware.

If there are multiple positioners and some of them are in an invalid state it’s possible to force upgrading the firmware to only certain positioners

jaeger upgrade-firmware -f -p 101 ~/Downloads/tendo_v04.00.04.bin

The shatterdome#

The CAN bus#

Loading from a profile#

The command queue#

Multibus interfaces#

The FPS class#

Initialisation#

Sending commands#

Shutting down the FPS#

FPS as a context manager#

Adding a single positioner#

Sending trajectories#

Aborting all trajectories#

Positioner, status, and position#

Positioner initialisation#

Position and status pollers#

Sending a positioner to a position#

Waiting for a status#

Commands#

Replies#

When is a command marked done?#

Time-outs#

Accessing the IEB#

Internals#

Configuration files#

Logging#

The bootloader mode#

Upgrading firmware#

The `CAN bus`#

The `FPS` class#

`Positioner`, status, and position#