Preparation: paasta_tools and yelpsoa-configs

paasta_tools reads configuration about services from several YAML files in soa-configs:

Each object inside of these YAML files is called an “instance” of a PaaSTA service. It describes a unique way to run the service, with a unique command, cpu and ram requirements, etc.

Duplication can be reduced by using YAML anchors and merges. PaaSTA will not attempt to run any definition prefixed with _, so you are free to use them for YAML templates.

Note that service names (the name of the folder where your config file is located) should be no more than 63 characters. For kubernetes services(config files with kubernetes as prefix), the instance names should be no more than 63 characters as well. _ is counted as two character. We convert _ to – because underscore is not allowed in kubernetes Pod names.

Example:

_template: &template
    env:
        foo: bar

main:
    <<: *template
    cpus: 1.0
    mem: 1000
    command: busybox httpd

worker:
    <<: *template
    cpus: 0.1
    mem: 100
    command: python -m worker

`Common Settings`

All configuration files that define something to launch on a PaaSTA Cluster can specify the following options:

cpus: Number of CPUs an instance needs. Defaults to 1. CPUs in Kubernetes are “shares” and represent a minimal amount of a CPU to share with a Pod relative to the other Pods on a host. For a more detailed read on how this works in practice, see the docs on isolation.

cpu_burst_add: Maximum number of additional CPUs an instance may use while bursting; if unspecified, PaaSTA defaults to 1 for long-running services, and 0 for scheduled jobs (Tron). For example, if a service specifies that it needs 2 CPUs normally and 1 for burst, the service may go up to 3 CPUs, if needed.

mem: Memory (in MB) an instance needs. Defaults to 4096 (4GB). In Kubernetes memory is constrained to the specified limit, and containers will reach out-of-memory (OOM) conditions if they attempt to exceed these limits, and then be killed. For more a more detailed read on how this works, see the docs on isolation

disk: Disk (in MB) an instance needs. Defaults to 1024 (1GB).
env: A dictionary of environment variables that will be made available to the container. Values can be plain strings or use the SECRET() macro to inject secrets from the service’s secrets/ directory. For example:
env:
  MY_SECRET_VAR: SECRET(test-secret)
  PAASTA_SECRET_MYSQL_VAR: SECRET(mysql_credentials)
  REGULAR_VAR: "plain-value"
The SECRET() macro references secrets that are stored encrypted in yelpsoa-configs and decrypted by Vault. The secret name corresponds to the filename in the service’s secrets/ directory without the .json suffix (e.g., SECRET(test-secret) references secrets/test-secret.json). Secrets are managed by the paasta secret command. For more details, see PaaSTA service secrets.

Secrets can also be injected as files. See secret_volumes.

PaaSTA additionally will inject the following variables automatically (keep in mind all environment variables are strings in a shell):

MESH_REGISTRATIONS (optional): space-separated list of registrations for a given Pod if the PaaSTA instance is mesh-routable - absent otherwise.

PAASTA_CLUSTER: The cluster name

PAASTA_DEPLOY_GROUP: The deploy group specified

PAASTA_DOCKER_IMAGE: The docker image name

PAASTA_GIT_SHA: The short git sha of the code the container has

PAASTA_HOST: The hostname of the actual server the container is running on

PAASTA_IMAGE_VERSION (optional) : The version of the docker image - only present for instances using automated redeploys or manually deployed using –image-version

PAASTA_INSTANCE_TYPE: The instance type of the service (e.g: tron, kubernetes, eks, etc)

PAASTA_INSTANCE: The instance name

PAASTA_LAUNCHED_BY (optional): May not be present. If present, will have the username of the user who launched the PaaSTA container

PAASTA_MONITORING_TEAM: The team that is configured to get alerts.

PAASTA_POD_IP: The IP of the Pod running the PaaSTA container - may not always be a routable IP

PAASTA_PORT: The configured port the service should listen on

PAASTA_RESOURCE_CPUS: Number of cpus allocated to a container

PAASTA_RESOURCE_DISK: Amount of disk space in MB allocated to a container

PAASTA_RESOURCE_GPUS: Number of GPUS (if requested) allocated to a container

PAASTA_RESOURCE_MEM: Amount of ram in MB allocated to a container

PAASTA_SERVICE: The service name

POD_NAME: The Pod name as generated by Kubernetes.
secret_volumes: An array of dictionaries for mounting secrets as files in the container. Secrets are stored encrypted in yelpsoa-configs and decrypted by Vault. Secrets are managed by the paasta secret command. There are two ways to configure secret volumes:

Simple format (secret name becomes the filename within the directory):
secret_volumes:
  - container_path: /nail/foo
    secret_name: the_secret_1
  - container_path: /nail/bar
    secret_name: the_secret_2
This creates files at /nail/foo/the_secret_1 and /nail/bar/the_secret_2. The container_path is the directory, and secret_name becomes the filename.

Items format (explicit filenames for multiple secrets in one directory):
secret_volumes:
  - container_path: /nail/foo
    items:
      - key: the_secret_1
        path: bar.yaml
      - key: the_secret_2
        path: baz.yaml
This creates files at /nail/foo/bar.yaml and /nail/foo/baz.yaml.

The simple format example above is equivalent to:
secret_volumes:
  - container_path: /nail/foo
    items:
      - key: the_secret_1
        path: the_secret_1
  - container_path: /nail/bar
    items:
      - key: the_secret_2
        path: the_secret_2
Each secret volume dictionary must contain container_path and either secret_name OR items:

container_path: The directory in the container where the secret file(s) will be mounted. (Required for both formats)

secret_name: (Simple format only) The name of the secret, corresponding to the filename in secrets/ without the .json suffix (e.g., test-secret references secrets/test-secret.json). The file will be created with this name. This field is ignored if items is specified.

items: (Items format only) A list of secrets with custom filenames. If specified, secret_name is ignored:

key: The name of the secret, corresponding to the filename in secrets/ without the .json suffix.

path: The filename to use in the container.

For more details, see PaaSTA service secrets.
extra_volumes: An array of dictionaries specifying extra bind-mounts inside the container. Can be used to expose filesystem resources available on the host into the running container. Common use cases might be to share secrets that exist on the host, or mapping read-write volumes for shared data location. For example:
extra_volumes:
  - {containerPath: /etc/secrets, hostPath: /etc/secrets, mode: RO}
  - {containerPath: /tmp, hostPath: /tmp, mode: RW}
Note: The format of these dictionaries must match the specification for the Mesos Docker containers schema, no error-checking is performed.

Note: In the case of a conflict between the extra_volumes and the system-configured volumes, extra_volumes will take precedence.

WARNING: This option should be used sparingly. Any specified bind-mount must exist on the filesystem beforehand, or the container will not run. Additionally it is possible for a service to be defined with a read-write volume on a sensitive part of the filesystem, as root. PaaSTA does not validate that the bind mounts are “safe”.
uses_bulkdata: A boolean indicating whether the service should mount the directory /nail/bulkdata on the host into the container. Defaults to true.

Placement Options

Placement options provide control over how PaaSTA schedules a task, whether it is scheduled by Kubernetes, Tron, or paasta remote-run. Most commonly, it is used to restrict tasks to specific locations.

General

These options are applicable to tasks scheduled through Kubernetes.

deploy_blacklist: A list of lists indicating a set of locations to not deploy to. For example:

deploy_blacklist: [["region", "uswest1-prod"]]

would indicate that PaaSTA should not deploy the service to the uswest1-prod region.

deploy_whitelist: A list of lists indicating a set of locations where deployment is allowed. For example:

deploy_whitelist: ["region", ["uswest1-prod", "uswest2-prod"]]

would indicate that PaaSTA can only deploy in uswest1-prod or uswest2-prod. If this list is empty (the default), then deployment is allowed anywhere. This is superseded by the blacklist; if a host is both whitelisted and blacklisted, the blacklist will take precedence. Only one location type of whitelisting may be specified.

pool: The pool of machines a PaaSTA app runs in. If no pool is set, an app will automatically be set to run in default pool.

Warning: In order for an service to be launched in a particular pool, there must exist some nodes that already exist with that particular pool attribute set.

Kubernetes

These options are only applicable to tasks scheduled on Kubernetes.

topology_spread_constraints: A set of rules to spread Pods across topologies, for example to try spreading Pods evenly across both nodes and availability zones:
topology_spread_constraints:
  - max_skew: 1
    topology_key: "topology.kubernetes.io/zone"
    when_unsatisfiable: "ScheduleAnyway"
  - max_skew: 1
    topology_key: "kubernetes.io/hostname"
    when_unsatisfiable: "ScheduleAnyway"
These can be configured per cluster (or globally) and will be added to every Pod Spec template, using paasta.yelp.com/service and paasta.yelp.com/instance as selectors.

To avoid conflicts with the deploy_whitelist and deploy_blacklist, please only use when_unsatisfiable: “ScheduleAnyway” (at least until PAASTA-17951 is resolved).

For more information, see the official Kubernetes documentation on topology spread constraints.
unhealthy_pod_eviction_policy: An option to control when unhealthy Pods can be evicted under a PodDisruptionBudget (PDB).

The default is AlwaysAllow, which essentially excludes unhealthy Pods from the PDB and allow eviction.

This can also be set to IfHealthyBudget which prevents disrupting unhealthy Pods if their eviction would violate the PDB constraints. For example:
unhealthy_pod_eviction_policy: IfHealthyBudget
This value is generally configured by PaaSTA globally, but can be overridden per-instance as a power-user override - you should generally not be configuring this yourself. For more information, see the official Kubernetes documentation on unhealthy pod eviction policy.
node_selectors: A map of labels a node is required to have for a task to be launched on said node. There are several ways to define a selector. The simplest is a key-value pair. For example, this selector restricts a task to c3.8xlarge instances:
node_selectors:
  instance_type: c3.8xlarge
The value can also be a list of multiple values. For example, this selector restricts a task to both c3.8xlarge and m5.2xlarge instances:
node_selectors:
  instance_type: ["c3.8xlarge", "m5.2xlarge"]
For more complex cases, an operator, and optionally a value (or values) must be set. Here are some examples:
Restricts a task to instances that are not c3.8xlarges:
node_selectors:
  instance_type:
    - operator: NotIn
      values: ["c3.8xlarge"]
Requires that a node have the ssd label:
node_selectors:
  ssd:
    - operator: Exists
Requires that a node not have the ssd label:
node_selectors:
  ssd:
    - operator: DoesNotExist
Requires that a node have a label priority with a value greater than 1 and less than 5:
node_selectors:
  priority:
    - operator: Gt
      value: 1
    - operator: Lt
      value: 5
Note

The label instance_type is special. If set as a node selector, PaaSTA will automatically convert it to a canonical version set by Kubernetes on all AWS nodes.
anti_affinity: A set of rules define when a node should not be selected for spawning a task in terms of task running on the node. This can be used to schedule a single task per node and provide better resource isolation for resource intensive tasks. For example:
anti_affinity:
  service: acron
for a service acron indicates to not schedule any 2 instances acron service on the same host. This can be extended to service instances also. For example:
anti_affinity:
  service: acron
  instance: test
would indicate to not schedule any 2 instances with name test of service acron on the same host. Multiple anti_affinities rules can also be used which will result AND-ing of all the rules. For example:
anti_affinity:
  - service: acron
  - service: kafka-k8s
would indicate the scheduler to not select a node when both acron and kafka-k8s is running on the node Note: anti_affinity rules should be used with judiciously and with caution as they require substantial processing and may slow down scheduling significantly in large clusters

For more information on selector operators, see the official Kubernetes documentation on node affinities.

pod_management_policy: An option for applications managed with StatefulSets to determine if the Pods are managed in parallel or in order.

The default value is OrderedReady. It can be set to Parallel. For example:
pod_management_policy: Parallel

`eks-[clustername].yaml`

Note: All values in this file except the following will cause PaaSTA to bounce the service:

min_instances
instances
max_instances
backoff_seconds

Top level keys are instance names, e.g. main and canary. Each instance MAY have:

Anything in the Common Settings.

Anything from General Placement Options and Kubernetes Placement Options.
cap_add: List of capabilities that are passed to Docker. Defaults to empty list. Example:
"cap_add": ["IPC_LOCK", "SYS_PTRACE"]
instances: Kubernetes will attempt to run this many instances of the Service

min_instances: When autoscaling, the minimum number of instances that kubernetes will create for a service. Defaults to 1.

max_instances: When autoscaling, the maximum number of instances that kubernetes will create for this service. If specified, instances is ignored.

registrations: A list of SmartStack registrations (service.namespace) where instances of this PaaSTA service ought register in. In SmartStack, each service has difference pools of backend servers that are listening on a particular port. In PaaSTA we call these “Registrations”. By default, the Registration assigned to a particular instance in PaaSTA has the same name, so a service foo with a main instance will correspond to the foo.main Registration. This would correspond to the SmartStack namespace defined in the Registration service’s smartstack.yaml. This registrations option allows users to make PaaSTA instances appear under an alternative namespace (or even service). For example canary instances can have registrations: ['foo.main'] to route their traffic to the same pool as the other main instances.

container_port: Specify the port to expose when in bridge mode. Defaults to 8888.

bounce_method: Controls the bounce method; see bounce_lib Note: the upthendown bounce is not available to kubernetes instances.

bounce_health_params: A dictionary of parameters for get_happy_tasks.

check_haproxy: Boolean indicating if PaaSTA should check the local haproxy to make sure this task has been registered and discovered (Defaults to True if service is in SmartStack)

min_task_uptime: Minimum number of seconds that a task must be running before we consider it healthy (Disabled by default)

haproxy_min_fraction_up: if check_haproxy is True, we check haproxy on up to 20 boxes to see whether a task is available. This fraction of boxes must agree that the task is up for the bounce to treat a task as healthy. Defaults to 1.0 – haproxy on all queried boxes must agree that the task is up.

bounce_overprovision_factor: controls the maximum number of extra instances allowed above the desired count during a bounce, as a fraction from 0.25 to 1. 0.25 is set as the floor to ensure at least 25% surge capacity is allowed. * Type: Number (0.25-1) * Default: System default is set to 1.0

Example:: bounce_overprovision_factor : 0.25

bounce_margin_factor: proportionally increase the number of old instances to be drained when the crossover bounce method is used. 0 < bounce_margin_factor <= 1. Defaults to 1 (no influence). This allows bounces to proceed in the face of a percentage of failures. It doesn’t affect any other bounce method but crossover. See the bounce docs for a more detailed description.

bounce_start_deadline: a floating point number of seconds to add to the deadline when deployd notices a change to soa-configs or the marked-for-deployment version of an instance. Defaults to 0. (deadline = now) When deployd has a queue of instances to process, it will choose to process instances with a lower deadline first. Set this to a large positive number to allow deployd to process other instances before this one, even if their

soa-configs change or mark-for-deployment happened after this one.

This setting only affects the first time deployd processes an instance after a change –
instances that need to be reprocessed will be reenqueued normally.

drain_method: Controls the drain method; see drain_lib. Defaults to noop for instances that are not in Smartstack, or hacheck if they are.

drain_method_params: A dictionary of parameters for the specified drain_method. Valid parameters are any of the kwargs defined for the specified bounce_method in drain_lib.

cmd: The command that is executed. If a string, will be wrapped in /bin/sh -c. If a list, will be executed directly as is with no shell parsing. WARNING: if the command registers any signal handlers (e.g., you want to gracefully shutdown upon SIGTERM), you’ll need to pass the command in as a list to ensure that Kubernetes sends signals to a process that can handle them. For internal users, COMPINFRA-4755 has some information around ways that we’ll make this automatic/unnecessary in the future.

args: An array of docker args if you use the “entrypoint” functionality.

monitoring: See the monitoring.yaml section for details.

autoscaling: See the autoscaling docs for details

metrics_providers: A list of data sources to use for autoscaling:

type: Which method the autoscaler will use to determine a service’s utilization. Should be cpu, worker-load, uwsgi, active-requests, piscina, gunicorn, or arbitrary-promql.

decision_policy: Which method the autoscaler will use to determine when to autoscale a service. Should be proportional or bespoke.

setpoint: The target utilization (as measured by your metrics_provider) that the autoscaler will try to achieve. Default value is 0.8.

desired_active_requests_per_replica: Only valid for the active-requests metrics provider. The target number of requests per second each Pod should be receiving.

max_instances_alert_threshold: If the autoscaler has scaled your service to max_instances, and the service’s utilization (as measured by your metrics_provider) is above this value, you’ll get an alert. The default is the same as your setpoint.

moving_average_window_seconds: A smoothing function to apply to the data received from your metrics provider.

metrics_query: (arbitrary-promql only) The PromQL query that returns your metric value.

series_query: (arbitrary-promql only, optional) Custom series query for metric discovery. If omitted, PaaSTA automatically generates one.

target_type: (arbitrary-promql only, optional) How the HPA interprets the metric value. Either AverageValue (default) or Value. See autoscaling for details.

resources: (arbitrary-promql only, optional, advanced) Custom prometheus-adapter resource mappings. Controls how Prometheus labels are mapped to Kubernetes resources.

scaledown_policies: Custom configuration for the Kubernetes HPA controlling when the service will scale down; this parameter exactly follows the Kubernetes HPA schema <https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#scaling-policies> for scaling policies.

deploy_group: A string identifying what deploy group this instance belongs to. The step parameter in deploy.yaml references this value to determine the order in which to build & deploy deploy groups. Defaults to clustername.instancename. See the deploy group doc_ for more information.

cost_owner: A string identifying the team responsible for the cost of this instance. Applied as a yelp.com/cost_owner Kubernetes pod label for cost attribution. Should be in kebab-case and match a canonical owner name from the Ownership service. Must match the pattern ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$ and be at most 63 characters. If not specified, inherits from service.yaml.

replication_threshold: An integer representing the percentage of instances that need to be available for monitoring purposes. If less than replication_threshold percent instances of a service’s backends are available, the monitoring scripts will send a CRITICAL alert.

healthcheck_mode: One of cmd, tcp, http, or https. If set to http or https, a curl command will be executed inside the container.

If set to cmd then PaaSTA will execute healthcheck_cmd and examine the return code. It must return 0 to be considered healthy.

If the service is registered in SmartStack, the healthcheck_mode will automatically use the same setings specified by smartstack.yaml.

If not in smartstack, the default healthcheck is “None”, which means the container is considered healthy unless it crashes.

A http healthcheck is considered healthy if it returns a 2xx or 3xx response code.

healthcheck_cmd: If healthcheck_mode is set to cmd, then this command is executed inside the container as a healthcheck. It must exit with status code 0 to signify a successful healthcheck. Any other exit code is treated as a failure. This is a required field if healthcheck_mode is cmd.

healthcheck_grace_period_seconds: Kubernetes will wait this long after the container has started before liveness probe is initiated. Defaults to 60 seconds. Readiness probes will always start after 10 seconds. The application should able to receive traffic as soon as the readiness probe is successful. Keep this in mind for any expensive “warm-up” requests.

A failing readiness probe will not restart the instance, it will however be removed from the mesh and not receive any new traffic.

To add an additional delay after the Pod has started and before probes should start, see min_task_uptime.

healthcheck_interval_seconds: Kubernetes will wait this long between healthchecks. Defaults to 10 seconds.

healthcheck_timeout_seconds: Kubernetes will wait this long for a healthcheck to return before considering it a failure. Defaults to 10 seconds.

healthcheck_max_consecutive_failures: Kubernetes will kill the current task if this many healthchecks fail consecutively. Defaults to 30 attempts.

healthcheck_uri: The url of the service to healthcheck if using http. Defaults to the same uri specified in smartstack.yaml, but can be set to something different here.

net: Specify which kind of networking mode adhoc containers of this service should be launched using. Defaults to 'bridge'.

prometheus_shard: Optional name of Prometheus shard to be configured to scrape the service. This shard should already exist and will not be automatically created.

prometheus_path: Optional path the Prometheus shard to be configured with to scrape the service. This shard should already exist and will not be automatically created.

prometheus_port: Optional port, not equal to container_port, to expose for prometheus scraping.

routable_ip: Optionally assign this instance a routable IP so it can be accessed externally. This option is implied when registered to smartstack or when specifying a prometheus_port. Defaults to false

weight: Load balancer/service mesh weight to assign to Pods belonging to this instance. Pods should receive traffic proportional to their weight, i.e. a Pod with weight 20 should receive 2x as much traffic as a Pod with weight 10. Defaults to 10. Must be an integer. This only makes a difference when some Pods in the same load balancer have different weights than others, such as when you have two or more instances with the same registration but different weight.

lifecycle: A dictionary of additional options that adjust the termination phase of the pod lifecycle:

pre_stop_command: The command to run in your container before stopping. This could handle gracefully stopping or checkpointing your worker, for example. This can be a list of strings (command + arguments) or a single string (which gets turned into a single-element list by Paasta). After the pre_stop_command is run, Kubernetes sends SIGTERM to the main process in the container. Your main process may then handle the SIGTERM signal and do any additional cleanup it needs to do before exiting. Most HTTP services at Yelp will shut down immediately when they receive SIGTERM (which can interrupt in-flight requests). Kew and SFN workers will finish their existing task before shutting down. If your shutdown behavior (pre_stop_command plus anything your main container does after receiving SIGTERM) takes longer than 30 seconds, you’ll want to specify termination_grace_period_seconds to be long enough. The default pre-stop command is configured by the next few options.

pre_stop_drain_seconds: For services registered in the mesh, we apply a default pre_stop_command which starts draining the service from the mesh and waits this many seconds before stopping the container. Defaults to 30 seconds for services that are registered in the mesh, and 0 seconds for services that are not.

pre_stop_wait_for_connections_to_complete: If set to true, the default pre_stop_command will:

Sleep pre_stop_drain_seconds seconds, to give the service mesh time to remove the pod from Envoy everywhere.

Wait for connections on container_port (8888 by default) to complete (checking once per second.)

This is useful for services with long-running requests that need to be completed before the container is stopped. Defaults to true if your service is registered in smartstack (the mesh). Otherwise, defaults to false. You may override this to false if you don’t want to wait for connections to complete. You may override this to true if you do want connections to complete, but your service is not registered in smartstack. Note that this will only wait for connections on container_port to complete. It does not wait for any outbound connections to complete, nor inbound connections on other ports. This is implemented with a shell one-liner that depends on /bin/sh and grep being available in the container. If your container does not have /bin/sh or grep, you will need to set this to false.

termination_grace_period_seconds: the number of seconds to allow pre-stop hooks to complete before forcibly killing your instance. Note that the instance will be forcibly killed after this period, so your pre_stop_command and any post-SIGTERM behavior should complete well within this time period!

If your service is registered in the mesh, this defaults to long enough for the default pre-stop hook defined by pre_stop_drain_seconds and pre_stop_wait_for_connections_to_complete to complete. Exception: if timeout_server_ms or endpoint_timeouts in your smartstack.yaml specifies a timeout greater than 1800000ms (30 minutes), the default will cap at 30 minutes. This is done under the assumption that most services with timeouts that long are able to handle SIGTERM to gracefully close connections. If your service really does need to maintain connections for longer than 30 minutes, and cannot gracefully close connections, just set termination_grace_period_seconds to an appropriate value.

If your service is not registered in the mesh, it defaults to the k8s default of 30s.

namespace: The Kubernetes namespace where Paasta will create objects related to this service. Defaults to paastasvc-service--name (that is, the service name will have underscores replaced with --.)
autotune_limits: Optionally set lower and upper-bounds for autotuned resources. Useful for services that have a known range of resource usage, but that have usage patterns that don’t play well with our autotune system. This is a dictionary with the following keys:

cpu: A dictionary with the keys min and max. These are the lower and upper bounds for the CPU limit, respectively.

mem: A dictionary with the keys min and max. These are the lower and upper bounds for the memory limit, respectively.

disk: A dictionary with the keys min and max. These are the lower and upper bounds for the disk limit, respectively.

Example:
autotune_limits:
  cpus:
    min: 0.1
  mem:
    max: 1024
  disk:
    min: 1000
    max: 10000
NOTE: it’s possible to set only one of the bounds, if you only want to set a lower or upper bound (i.e., setting both min/max is not required).
enable_aws_lb_readiness_gate: A boolean indicating whether to enable injecting AWS Load Balancer readiness gates for this instance. Defaults to false. This is only applicable to a very small subset of services at Yelp - please chat with #paasta before enabling this.

iam_role: The IAM role to be used by this service. If not specified, no IAM role will be used for Pod Identity.

boto_keys: A list of boto_cfg keys to be made available in the container’s environment. Use of this option is discouraged in favor of using IAM roles, but it is still necessary for some legacy services. If specified, only the listed boto_cfg keys will be made available in the container’s environment. Otherwise, the status-quo behavior will be used. See y/users-to-role for more information on how to switch to IAM roles. NOTE: this key is ignored for paasta local-run.

ssm_secrets: A list of configurations for syncing secrets from the AWS SSM Parameter Store to the service as environment variables. Each entry requires:

source: The name of the parameter in AWS SSM (e.g., /my-service/prod/db_password).

secret_name: The name of the environment variable to be created in the service.

Note: Fetching parameters requires the paasta-secrets-sync IAM role. The AWS region is automatically inferred from the target cluster’s configuration.

Note: Although many of these settings are inherited from smartstack.yaml, their thresholds are not the same. The reason for this has to do with control loops and infrastructure stability. The load balancer tier can be pickier about which copies of a service it can send requests to, compared to Mesos.

A load balancer can take a container out of service and put it back in a few seconds later. Minor flaps and transient errors are tolerated.

The healthchecks specified here in this file signal to the infrastructure that a container is unhealthy, and the action to take is to completely destroy it and launch it elsewhere. This is a more expensive operation than taking a container out of the load balancer, so it justifies having less sensitive thresholds.

`tron-[clustername].yaml`

This file stores configuration for periodically scheduled jobs for execution on Tron.

The documentation here is for the PaaSTA-specific options. For all other settings, please see the canonical docs.

Example Job

---
convert_logs:
  node: paasta
  schedule:
    start_time: 04:00:00
  actions:
    verify_logs_present:
      command: "ls /var/log/app/log_%(shortdate-1)s.txt"
      executor: ssh
    convert_logs:
      requires: [verify_logs_present]
      command: "convert_logs /var/log/app/log_%(shortdate-1)s.txt /var/log/app_converted/log_%(shortdate-1)s.txt"
      service: test_service
      deploy_group: prod
      cpus: .5
      mem: 100

PaaSTA-Specific Options

Each Tron job configuration MAY specify the following options:

deploy_group: A string identifying what deploy group this instance belongs to. The step parameter in deploy.yaml references this value to determine the order in which to build & deploy deploy groups. See the deploy group doc_ for more information.

service: Configures PaaSTA to use the docker image from an alternative service. This setting picks the default service for the whole job, but service may also be set on a per-action basis. Tron jobs may be composed of multiple actions that use commands from multiple different services.

monitoring: See the monitoring.yaml section for details.

cost_owner: A string identifying the team responsible for the cost of this job. Applied as a yelp.com/cost_owner Kubernetes pod label for cost attribution. Should be in kebab-case and match a canonical owner name from the Ownership service. Inherited by all actions unless overridden at the action level. If not specified, inherits from service.yaml. Must match the pattern ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$ and be at most 63 characters.

Each Tron action of a job MAY specify the following:

Anything in the Common Settings.

Anything from General Placement Options

service: Uses a docker image from different service. When service is set for an action, that setting takes precedence over what is set for the job.

executor: Configures Tron to execute the command in a particular way. Set to paasta to configure Tron to launch the job on the PaaSTA cluster. Defaults to paasta. When executor is NOT paasta (and is using ssh), all of these paasta-specific options listed here in this documentation will have no effect. It is OK to have a job composed of mixed paasta and ssh actions.

deploy_group: Same setting as the Job, but on a per-action basis. Defaults to the setting for the entire job.

cost_owner: Same setting as the Job, but on a per-action basis. Defaults to the setting for the entire job.

command: The command to run. If the action is configured with executor: paasta (default), then the command should be something available in the docker container (it should NOT start with paasta local-run).

ssm_secrets: A list of configurations for syncing secrets from the AWS SSM Parameter Store to the service as environment variables. Each entry requires:

source: The name of the parameter in AWS SSM (e.g., /my-service/prod/db_password).

secret_name: The name of the environment variable to be created in the service.

Note: Fetching parameters requires the paasta-secrets-sync IAM role. The AWS region is automatically inferred from the target cluster’s configuration.

If a Tron action of a job is of executor type spark, it MAY specify the following:

spark_paasta_cluster: The Paasta cluster on which to run spark jobs (spark executors). Default to the same cluster the tron job (spark driver) is running on.

spark_paasta_pool: The Paasta pool on which to run spark jobs (spark executors). Default to batch pool if not specified.

spark_args: Dictionary of spark configurations documented in https://spark.apache.org/docs/latest/configuration.html. Note some configurations are non- user-editable as they will be populated by PaaSTA. See https://github.com/Yelp/service_configuration_lib/blob/master/service_configuration_lib/spark_config.py#L9 for a complete list of such configurations.

aws_credentials_yaml: Path to the yaml file containing credentials to be set in the task’s AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables. Default to /etc/boto_cfg/<your-service-name>.yaml. If the file path does not exist, or the file does not contain keys for aws_access_key_id and aws_secret_access_key, those variables will be unset.

`adhoc-[clustername].yaml`

The yaml where adhoc instances are defined. Top-level keys are instance names. Each instance MAY have:

Anything in the Common Settings.

net

cmd

args

deploy_group

node_selectors

node_selectors_preferred

See the `kubernetes-[clustername].yaml`_ section for details for each of these parameters.

Footnotes:

`smartstack.yaml`

Configure service registration, discovery, load balancing and error alerting.

Here is an example smartstack.yaml:

---
main:
  extra_advertise:
    ecosystem:stagec
    - ecosystem:stagec
    region:uswest1-prod
    - region:uswest2-prod
  proxy_port: 20028
  timeout_server_ms: 5000

The main key is the service namespace. Namespaces were introduced for PaaSTA services in order to support running multiple daemons from a single service codebase. In PaaSTA, each instance in your kubernetes.yaml maps to a smartstack namespace of the same name, unless you specify a different registrations.

We now describe which keys are supported within a namespace. Note that all but proxy_port are optional.

proxy_port: integer-valued (or null) port on which HAProxy listens for requests. If null this service will be “discovery only” meaning that it will generate synapse discovery files on every host, but no listening port will be allocated. This must be unique across all environments where PaaSTA (or synapse) runs. At Yelp, we pick from the range [19000, 21000]. Feel free to pick the next available value – paasta fsm will do this for you automatically!

mode: string of value http or tcp, specifying whether the service is an HTTP or TCP service respectively. Defaults to http.
extra_headers: for use in http mode. Headers that should be added to the request before being forwarded to the server. Example:
extra_headers:
  X-Mode: ro
proxied_through: for specifying a service that will proxy requests transparently. This can be used to, for example, to proxy requests to caching services.

The value of this directive needs to be another smartstack-

namsespace, for example, servicename.main

If the proxy service is down, then smartstack will automatically failover

to the default backend

Example:

proxied_through: servicename.main

The following three keys have to do with how your service instances advertise and how other services discover your service. If you want to change these you should understand how PaaSTA environments are laid out and classified into latency zones. For information on this, see the environment_tools documentation.

advertise: List of location types that must be defined in your location_types.json. This is the level of your location hierarchy that your service instances advertise themselves at, and should match up with how your service is deployed. Think of this as the answer to “where does nerve register me”. Defaults to ['region']

If you set this to a wider location type than where your ZooKeeper clusters are deployed, this will cause additional ZooKeeper load. Be careful, and consult with the team responsible for your ZooKeeper clusters.

This is a list so that you can do a graceful change in discover (see immediately below). You add both types to the advertise list and then when you’ve switched discover you shrink it back down to one.

discover: Location type that SmartStack should discover your service instances at. Think of this as the answer to “where does synapse look for registrations”. Defaults to ‘region’.

This must be one of the location types listed in your advertise key to make sense.

Changing this key has potential to bring down your service everywhere all at once, change it only if you are sure what you are doing.
extra_advertise: Mapping of location to locations, where locations are specified as <location_type>:<location_instance> strings. For example, habitat:devc or ecosystem:prod. These are ad-hoc advertisements that read logically left to right. If you have X: [Y] in your extra_advertise object, then that will cause service instances in X to be available in Y. Defaults to an empty dictionary.
Be careful when putting a large location on the right side of this mapping: your service instances may be advertised into multiple ZooKeeper clusters.

The right-hand side must be the same or more general than your discover level. For instance, if your service discovers at the region level, you cannot advertise to a habitat, but you could advertise to a region or superregion.
Here’s an example of how to advertise an sf-devc service instance into uswest1-devc:
extra_advertise:
    region:sf-devc: ['region:uswest1-devc']

You can control your healthchecks with the following keys.

healthcheck_mode: specifies the mode for the healthcheck (http or tcp). Defaults to the mode of the service.

healthcheck_port: an alternative port to use for healthchecking your service. This is not required; it defaults to the port your service instance is running on.

healthcheck_uri: string specifying the URI which SmartStack should use to healthcheck the service. Defaults to /status. This is ignored if healthcheck_mode is tcp.

healthcheck_timeout_s: maximum number of seconds that a nerve healthcheck may take; defaults to 1.

updown_timeout_s: maximum number of seconds after which a service instance needs to be able to respond to healthchecks after updown_service has been called.
extra_healthcheck_headers: Extra headers/values that are appended to the healthcheck requests. Example:
extra_healthcheck_headers:
    X-Mode: ro

retries: Number of retries (only applies to connection failures), defaults to 1.

timeout_connect_ms: Envoy cluster connect_timeout in milliseconds, defaults to 200.

timeout_server_ms: Envoy route timeout in milliseconds, defaults to 1000.

idle_timeout: Envoy route idle_timeout in seconds, defaults to 60.

lb_policy: Envoy lb_policy
Defaults to “ROUND_ROBIN”.
endpoint_timeouts: Allows you to specify non-default server timeouts for specific endpoints. This is useful for when there is a long running endpoint that requires a large timeout value but you would like to keep the default timeout at a reasonable value.

Endpoints use prefix-matching by default; for example /specials/bulk/v1 will match both /specials/bulk/v1/foo and /specials/bulk/v1/bar.

Endpoints can also use regex matching, provided that the regex string begins with a caret ^ and backslashes within the string are properly escaped. For example, ^/specials/[^/]+/v2/\\d will match the endpoints /specials/bulk/v2/1 and ^/specials/milk/v2/2.

Example:
endpoint_timeouts:
    "/specials/bulk/v1": 15000
    "^/specials/[^/]+/v2/\\d": 11000

These keys are meant to control the built-in fault and delay injection features of Envoy, the new proxy we are using to replace HAProxy, and thus won’t work with the latter.

Note: The described Fault Injection features are currently only available internally at Yelp.

fixed_delay: A map of locations to delays. Controls the injection of a particular delay for a particular service, in a particular environment and for a particular percentage of requests. From the example:
main:
  proxy_port: 20028
  timeout_server_ms: 5000
  fixed_delay:
    ecosystem:stagec:
      duration_ms: 3000
      percent: 20
    superregion:uswest1-prod:
      duration_ms: 1000
      percent: 40
duration_ms: The duration of the delay in milliseconds.

percent: Percentage of requests (0-100) to be randomly affected by this delay. Note that due to Envoy’s current definition of percentage as an integer, this cannot be specified as a floating-point number.

Error Alerting

These keys provide optional overrides for the default alerting behaviour.

monitoring: Override default alerting behaviour. For example:
main:
  monitoring:
    team: frontend
    slack_channel: "notifications"
    project: "FRONTEND"
    ticket: true
    page: false
    page_nonprod: true
    error_threshold_ratio: 0.02
    minimum_error_rps: 10
    timeout_threshold_ratio: 0.8
    default_endpoint_alerting: true
    endpoints:
      - name: GET /something
      - name: GET /something/else
        error_threshold_ratio: 0.05
      - name: PUT /something
        error_threshold_ratio: 0.01
team: Override the default team for alerting.

slack_channel: Error alerts notify the first channel in the monitoring.yaml slack_channels list. Use this key if you prefer a different channel.

project: Override the default JIRA project for alerting.

ticket: Override the default ticketing behaviour. Error Alert ticketing defaults to false but also respects the ticketing behaviour set in the monitoring.yaml file. Override that here if required.

page: Override the default paging behaviour. Error Alert paging defaults to true but also respects the paging behaviour set in the monitoring.yaml file. Override that here if required.

page_nonprod: Override the default paging behaviour for non-production environments. Defaults to false.

error_threshold_ratio: Error threshold ratio (0-1) for errors under this namespace. Defaults to 0.01.

minimum_error_rps: Minimum error rate per second for errors under this namespace before an alert can be triggered, minimum is zero. Defaults to 5.

timeout_threshold_ratio: Ratio (0-1) of timeout_server_ms at which to alert on p99 latency. Defaults to 0.8 (80%).

timeout_alerting: Enable latency/timeout alerting for this namespace. When true, alerts fire to the configured slack_channel when p99 latency exceeds the threshold. Defaults to false.

timeout_page: Page on latency threshold breach. Only applies when timeout_alerting is true. Defaults to false.

timeout_ticket: Create a Jira ticket on latency threshold breach. Only applies when timeout_alerting is true. Defaults to false.

default_endpoint_alerting: Turn on alerts for all endpoints in this namespace. Defaults to false.

endpoint_error_threshold_ratio: Error threshold ratio (0-1) for errors to any singular endpoint. Defaults to the namespace error_threshold_ratio if specified, or 0.01.

endpoint_minimum_error_rps: Minimum error rate per second for errors to any singular endpoint before an alert can be triggered for errors to any singular endpoint. Defaults to the namespace minimum_error_rps if specified, or 5.

endpoints: List of endpoints to create specific alerts for.

name: The name of the endpoint.

error_threshold_ratio: Error threshold ratio (0-1). If not specified the threshold will be inherited from the endpoint_error_threshold_ratio; if that is not specified then the namespace’s error_threshold_ratio; otherwise 0.01.

minimum_error_rps: Minimum error rate per second for the endpoint. Minimum is zero. If not specified the threshold will be inherited from the endpoint_minimum_error_rps; if that is not specified then the namespace’s minimum_error_rps; otherwise 5.

Moving a Service to a different location type

If you don’t care about dropping traffic you can just change discover and advertise and then wait until the configuration and registrations propagate. If you want to do it gracefully you have to ensure that nerve registrations have time to propagate before you switch synapse’s discover key.

An example of switching from region to superregion discovery:

1a. Append to your advertise key:

- advertise: [region]
+ advertise: [region, superregion]

2. (Optional) Use zkCli.sh to monitor your new registrations for each superregion you are changing:

$ /usr/share/zookeeper/bin/zkCli.sh
[zk: localhost:2181(CONNECTING) 0] connect 10.40.5.6:22181
[zk: 10.40.5.6:22181(CONNECTED) 1] ls /nerve/superregion:norcal-devc/servicename.main
[host1-uswest1adevc_0000015910, host2-uswest1cdevc_0000015898, host3-uswest1cdevc_0000015893]
[zk: 10.40.5.6:22181(CONNECTED) 2]

2b. Run paasta status -v to verify that PaaSTA has balanced services across the infrastructure as expected.

Once zookeeper shows the proper servers, switch the discovery key:
```
- discover: region
+ discover: superregion
```
Wait a while, usually about 10 minutes, Then change advertise to just superregion:
```
- advertise: [region, superregion]
+ advertise: [superregion]
```

`monitoring.yaml`

The yaml where monitoring for the service is defined.

Defaults for a team can be set globally with the global Sensu configuration. team is the only mandatory key, but overrides can be set for the entire service with monitoring.yaml.

Additionally these settings can be overridden on a per-instance basis via the monitoring option. For example a canary instance can be set with page: false and team: devs, while the main instance can bet set to page: true and team: ops, and the dailyadsjob instance can be set with ticket: true and team: ads. See the Examples section for more examples.

Here is a list of options that PaaSTA will pass through:

team: Team that will be notified by Sensu.

page: Boolean to indicate if an instance should page if it is failing. Defaults to false.

runbook: An optional but highly recommended field. Try to use shortlinks when possible as sometimes the runbook url may need to be copied from a small screen.

tip: An optional one-line version of the runbook to help with common issues. For example: “Check to see if it is Bing first!”

notification_email: String representing an email address to send notifications to. This will default to the team email address if is is already set globally. For multiple emails, use a comma separated list.

irc_channels: Array of irc_channels to post notifications to.

slack_channels: Array of slack_channels to post notifications to.

ticket: Boolean to indicate if an alert should make a JIRA ticket. Defaults to false.

project: String naming the project where JIRA tickets will be created. Overrides the global default for the team.

priority: A JIRA ticket priority to use. This value should be a string value like '0', '1', '3.14', etc. If not set, the default will be the default_priority setting for the sensu team or the default priority used for the JIRA project.

tags: An list of tags that are used as labels when creating a JIRA ticket. Note that this list of tags does not overwrite the default values added for sensu checks (tags like SENSU for example), it just adds to that existing list.

component: Array of components affected by this check. These are used as components when creating a JIRA ticket.

description: A description giving more context on the check or event. This should be a longer expansion of information than what is included in the tip option.

alert_after: Time string that represents how long a a check should be failing before an actual alert should be fired. Currently defaults to 10m for the replication alert. See pysensu-yelp documentation<https://pysensu-yelp.readthedocs.io/en/latest/#pysensu_yelp.human_to_seconds> for accepted formats.

realert_every: An integer (not a time unit) representing how many checks to execute between sending alerts. The default settings is -1, which is a special value indicating exponential backoff. For example, given a check that runs once per minute, a setting of -1 would fire alerts on minutes 1,2,4,8,16, etc.

A setting of 60 would fire events on minutes 1, 61, 121, etc.

How often alerts are actually sent out is a function of this realert_every setting and the frequency at which a check runs, which in PaaSTA is once per minute.

check_every: Not a setting that can be configured!! This setting is described for completeness. In PaaSTA the check interval is not user-configurable and is hard-coded at 1m.

check_oom_events: Boolean to indicate if an instance should alert when the Out Of Memory killer kills processes in the instance containers. This alert sends an email to notification_email and post notifications to irc_channels. It does not page. Defaults to true.

Monitoring Examples

An example of a service that only pages on a cluster called “prod”:

# monitoring.yaml
team: devs
page: false

# kubernetes-prod.yaml
main:
  instances: 3
  monitoring:
     page: true

A service that pages everywhere, but only makes a ticket for a tron job:

# monitoring.yaml
team: backend
page: true

# tron-prod.yaml
nightly_batch:
  schedule: .....
  monitoring:
    page: false
    ticket: true

A kubernetes service that overrides options on different instances (canary):

# monitoring.yaml
team: frontend
page: false

# kubernetes-prod.yaml
main:
  instances: 20
  monitoring:
    team: ops
    page: true
canary:
  instances: 1
  registrations: ['service.main']
  monitoring:
    page: false
    ticket: true

A tron job that pages for a specific job in a service:

# monitoring.yaml
team: midend
page: false

# tron-prod.yaml
jobs:
  foo:
    monitoring:
      page: true

`service.yaml`

Various PaaSTA utilities look at the following keys from service.yaml

git_url

description

external_link

docker_registry This is optional. Set this to override the system-wide docker registry, and specify an alternate docker registry for your service.

cost_owner A string identifying the team responsible for the cost of this service. Applied as a yelp.com/cost_owner Kubernetes pod label for cost attribution. Should be in kebab-case and match a canonical owner name from the Ownership service. Must match the pattern ^[a-z0-9]([a-z0-9-]*[a-z0-9])?$ and be at most 63 characters. Can be overridden per-instance in instance configs or per-job/action in Tron configs.

Where does paasta_tools look for yelpsoa-configs?

By default, paasta_tools uses the system yelpsoa-configs dir, /nail/etc/services. Scripts should allow this to be overridden with -d or --soa-dir. Normally you would only do this for testing or debugging.