Migrating from Pywr v1.x

This guide is intended to help users of Pywr v1.x migrate to Pywr v2.x. Pywr v2.x is a complete rewrite of Pywr with a new API and new features. This guide will help you update your models to this new version.

Overview of the process

Pywr v2.x includes a more strict schema for defining models. This schema, along with the pywr-v1-schema crate, provide a way to convert models from v1.x to v2.x. However, this process is not perfect and will more than likely require manual intervention to complete the migration. The migration of larger and/or more complex models will require an iterative process of conversion and testing.

The overall process will follow these steps:

1. Convert the JSON from v1.x to v2.x using the provided conversion tool.
2. Handle any errors or warnings from the conversion tool.
3. Apply any other manual changes to the converted JSON.
4. (Optional) Save the converted JSON as a new file.
5. Load and run the new JSON file in Pywr v2.x.
6. Compare model outputs to ensure it behaves as expected. If necessary, make further changes to the above process and repeat.

Converting a model

The example below is a basic script that demonstrates how to convert a v1.x model to v2.x. This process converts the model at runtime, and does not replace the existing v1.x model with a v2.x definition.

Note: This example is meant to be a starting point for users to build their own conversion process; it is not a complete generic solution.

The function in the listing below is an example of the overall conversion process. The function takes a path to a JSON file containing a v1 Pywr model, and then converts it to v2.x.

1. The function reads the JSON, and applies the conversion function (convert_model_from_v1_json_string).
2. The conversion function that takes a JSON string and returns a tuple of the converted JSON string and a list of errors.
3. The function then handles these errors using the handle_conversion_error function.
4. After the errors are handled other arbitrary changes are applied using the patch_model function.
5. Finally, the converted JSON can be saved to a new file and run using Pywr v2.x.

from pywr import (
    convert_model_from_v1_json_string,
    ComponentConversionError,
    ConversionError,
    Schema,
)


def convert(v1_path: Path):
    with open(v1_path) as fh:
        v1_model_str = fh.read()
    # 1. Convert the v1 model to a v2 schema
    schema, errors = convert_model_from_v1_json_string(v1_model_str)

    schema_data = json.loads(schema.to_json_string())
    # 2. Handle any conversion errors
    for error in errors:
        handle_conversion_error(error, schema_data)

    # 3. Apply any other manual changes to the converted JSON.
    patch_model(schema_data)

    schema_data_str = json.dumps(schema_data, indent=4)
    # 4. Save the converted JSON as a new file (uncomment to save)
    # with open(v1_path.parent / "v2-model.json", "w") as fh:
    #     fh.write(schema_data_str)
    print("Conversion complete; running model...")
    # 5. Load and run the new JSON file in Pywr v2.x.
    schema = Schema.from_json_string(schema_data_str)
    model = schema.build(Path(__file__).parent, None)
    model.run("clp")
    print("Model run complete 🎉")

Handling conversion errors

The convert_model_from_v1_json_string function returns a list of errors that occurred during the conversion process. These errors can be handled in a variety of ways, such as modifying the model definition, raising exceptions, or ignoring them. It is suggested to implement a function that can handle these errors in a way that is appropriate for your use case. Begin by matching a few types of errors and then expand the matching as needed. By raising exceptions for unhandled errors, you can ensure that all errors are eventually accounted for, and that new errors are not missed.

The example handles the ComponentConversionError by matching on the error subclass (either Parameter() or Node()), and then handling each case separately. These two classes will contain the name of the component and optionally the attribute that caused the error. In addition, these types contain an inner error (ConversionError) that can be used to provide more detailed information. In the example, the UnrecognisedType() class is handled for Parameter() errors by applying the handle_custom_parameters function.

This second function adds a Pywr v2.x compatible custom parameter to the model definition using the same name and type (class name) as the original parameter.

def handle_conversion_error(error: ComponentConversionError, schema_data):
    """Handle a schema conversion error.

    Raises a `RuntimeError` if an unhandled error case is found.
    """
    match error:
        case ComponentConversionError.Parameter():
            match error.error:
                case ConversionError.UnrecognisedType() as e:
                    print(
                        f"Patching custom parameter of type {e.ty} with name {error.name}"
                    )
                    handle_custom_parameters(schema_data, error.name, e.ty)
                case _:
                    raise RuntimeError(f"Other parameter conversion error: {error}")
        case ComponentConversionError.Node():
            raise RuntimeError(f"Failed to convert node `{error.name}`: {error.error}")
        case _:
            raise RuntimeError(f"Unexpected conversion error: {error}")


def handle_custom_parameters(schema_data, name: str, p_type: str):
    """Patch the v2 schema to add the custom parameter with `name` and `p_type`."""

    # Ensure the network parameters is a list
    if schema_data["network"]["parameters"] is None:
        schema_data["network"]["parameters"] = []

    schema_data["network"]["parameters"].append(
        {
            "meta": {"name": name},
            "type": "Python",
            "source": {"path": "v2_custom_parameter.py"},
            "object": p_type,  # Use the same class name in v1 & v2
            "args": [],
            "kwargs": {},
        }
    )

Other changes

The upgrade to v2.x may require other changes to the model. For example, the conversion process does not currently handle recorders and other model outputs. These will need to be manually added to the model definition. Such manual changes can be applied using, for example a patch_model function. This function will make arbitrary changes to the model definition. The example, below updates the metadata of the model to modify the description.

def patch_model(schema_data):
    """Patch the v2 schema to add any additional changes."""
    # Add any additional patches here
    schema_data["metadata"]["description"] = "Converted from v1 model"

Full example

The complete example below demonstrates the conversion process for a v1.x model to v2.x:

import json
from pathlib import Path

# ANCHOR: convert
from pywr import (
    convert_model_from_v1_json_string,
    ComponentConversionError,
    ConversionError,
    Schema,
)


def convert(v1_path: Path):
    with open(v1_path) as fh:
        v1_model_str = fh.read()
    # 1. Convert the v1 model to a v2 schema
    schema, errors = convert_model_from_v1_json_string(v1_model_str)

    schema_data = json.loads(schema.to_json_string())
    # 2. Handle any conversion errors
    for error in errors:
        handle_conversion_error(error, schema_data)

    # 3. Apply any other manual changes to the converted JSON.
    patch_model(schema_data)

    schema_data_str = json.dumps(schema_data, indent=4)
    # 4. Save the converted JSON as a new file (uncomment to save)
    # with open(v1_path.parent / "v2-model.json", "w") as fh:
    #     fh.write(schema_data_str)
    print("Conversion complete; running model...")
    # 5. Load and run the new JSON file in Pywr v2.x.
    schema = Schema.from_json_string(schema_data_str)
    model = schema.build(Path(__file__).parent, None)
    model.run("clp")
    print("Model run complete 🎉")


# ANCHOR_END: convert
# ANCHOR: handle_conversion_error
def handle_conversion_error(error: ComponentConversionError, schema_data):
    """Handle a schema conversion error.

    Raises a `RuntimeError` if an unhandled error case is found.
    """
    match error:
        case ComponentConversionError.Parameter():
            match error.error:
                case ConversionError.UnrecognisedType() as e:
                    print(
                        f"Patching custom parameter of type {e.ty} with name {error.name}"
                    )
                    handle_custom_parameters(schema_data, error.name, e.ty)
                case _:
                    raise RuntimeError(f"Other parameter conversion error: {error}")
        case ComponentConversionError.Node():
            raise RuntimeError(f"Failed to convert node `{error.name}`: {error.error}")
        case _:
            raise RuntimeError(f"Unexpected conversion error: {error}")


def handle_custom_parameters(schema_data, name: str, p_type: str):
    """Patch the v2 schema to add the custom parameter with `name` and `p_type`."""

    # Ensure the network parameters is a list
    if schema_data["network"]["parameters"] is None:
        schema_data["network"]["parameters"] = []

    schema_data["network"]["parameters"].append(
        {
            "meta": {"name": name},
            "type": "Python",
            "source": {"path": "v2_custom_parameter.py"},
            "object": p_type,  # Use the same class name in v1 & v2
            "args": [],
            "kwargs": {},
        }
    )


# ANCHOR_END: handle_conversion_error
# ANCHOR: patch_model
def patch_model(schema_data):
    """Patch the v2 schema to add any additional changes."""
    # Add any additional patches here
    schema_data["metadata"]["description"] = "Converted from v1 model"


# ANCHOR_END: patch_model

if __name__ == "__main__":
    pth = Path(__file__).parent / "v1-model.json"
    convert(pth)

Converting custom parameters

The main changes to custom parameters in Pywr v2.x are as follows:

1. Custom parameters are no longer required to be a subclass of Parameter. They instead can be simple Python classes that implement a calc method.
2. Users are no longer required to handle scenarios within custom parameters. Instead an instance of the custom parameter is created for each scenario in the simulation. This simplifies writing parameters and removes the risk of accidentally contaminating state between scenarios.
3. Custom parameters are now added to the model using the "Python" parameter type. I.e. the "type" field in the parameter definition should be set to "Python" (not the class name of the custom parameter). This parameter type requires that the user explicitly define which metrics the custom parameter requires.

Simple example

v1.x custom parameter:

from pywr.parameters import ConstantParameter


class MyParameter(ConstantParameter):
    def value(self, *args, **kwargs):
        return 42


MyParameter.register()

v2.x custom parameter:

class MyParameter:
    def calc(self, *args, **kwargs):
        return 42