Cooking: How it all comes together¶
Ways That PaaSTA Can Run Services¶
Long Running Services¶
Long running services are are processes that are expected to run continuously and usually have the same process id throughout. PaaSTA uses Marathon to configure how these services should run.
These services often serve network traffic, usually HTTP. PaaSTA integrates with SmartStack to make it easy for others to discover these long running services, or operators can use alternative service discovery methods to interact with them.
Alternative names: workers, daemons, batch daemons, servers
Scheduled tasks are those tasks that are periodically run, and are not expected to run continuously. Due to their ephemeral nature, they often do not expose a TCP port.
You can schedule jobs on PaaSTA using Tron.
Adhoc tasks are often required to support one-time tasks, like a database
migration. Sometimes they are also useful for exploratory purposes, or even
sometimes for end-to-end tests. PaaSTA supports this use case through
local-run, which supports building images locally, as well as using the same
image as a “live” deployment.
See the adhoc instances doc for more information on
running adhoc instances using
Alternative names: Adhoc batches, interactive batches, one-off dynos, one-off tasks
soa-configs are used to centrally configure services. See the documentation for
soa-configs for more information on what these are.
Contract with services¶
The Paasta Contract describes the responsibilities of services that wish to work with PaaSTA.
paasta_tools contains the implementation of several of these rules. For example, generate_deployments_for_service is the piece that checks each service’s git repo for the specially-named branch that tells PaaSTA which versions of the service should go to which clusters.
A yelpsoa-configs master runs generate_deployments_for_service
frequently. The generated
deployments.json appears in
/nail/etc/services/service_name throughout the cluster.
Marathon masters run deploy_marathon_services,
a thin wrapper around
These scripts parse
deployments.json and the current cluster state,
then issue commands to Marathon to put the cluster into the right state
– cluster X should be running version Y of service Z.
How PaaSTA Runs Docker Containers¶
Marathon launches the Docker containers that comprise a PaaSTA service.
Docker images are run by Mesos’s native Docker executor. PaaSTA composes the configuration for the running image:
--attach: stdout and stderr from running images are sent to logs that end up in the Mesos sandbox (currently unavailable).
--cpu-shares: This is the value set in
--memory: This is the value set in
--memory-swap: Total memory limit (memory + swap). We set this to the same value as “mem”, rounded up to the nearest MB, to prevent containers being able to swap.
--net: PaaSTA uses bridge mode to enable random port allocation.
--env: Any environment variables specified in the
envsection will be here. Additional
MESOS_environment variables will also be injected, see the related docs for more information.
--publish: Mesos picks a random port on the host that maps to and exposes port 8888 inside the container. This random port is announced to Smartstack so that it can be used for load balancing.
--privileged: Containers run by PaaSTA are not privileged.
--restart: No restart policy is set on PaaSTA containers. Restarting tasks is left as a job for the Framework (Marathon).
--rm: Mesos containers are rm’d after they finish.
--tty: Mesos containers are not given a tty.
--volume: Volume mapping is controlled via the paasta_tools configuration. PaaSTA uses the volumes declared in
/etc/paasta/volumes.jsonas well as per-service volumes declared in
extra_volumesdeclared in the soa-configs.
--workdir: Mesos containers are launched in a temporary “workspace” directory on disk. Use the workdir sparingly and try not to output files.
Mesos is the actual system that runs the docker images. In Mesos land these are called “TASKS”. PaaSTA-configured tasks use exponential backoff to prevent unhealthy tasks from continuously filling up disks and logs – the more times that your service has failed to start, the longer Mesos will wait before trying to start it again.
Mesos will healthcheck the task based on the same healthcheck that SmartStack uses, in order to prune unhealthy tasks. This pruning is less aggressive than SmartStack’s checking, so a dead task will go DOWN in SmartStack before it is reaped by Marathon. By default the healthchecks occur every 10 seconds, and a service must fail 30 times before that task is pruned and a new one is launched in its place. This means a task had 5 minutes by default to properly respond to its healthchecks.
Time Zones In Docker Containers¶
Docker containers launched by PaaSTA have whatever time zone is set by the Dockerfile. If it is not set, the default is the Linux default, UTC.
Some code makes assumptions about the underlying time zone a server is in. In such a situation the time zone should be explicitly set in the Dockerfile. For example, this line can be placed in a Dockerfile to set the container to run in US Pacific time:
RUN ln -fs /usr/share/zoneinfo/US/Pacific /etc/localtime
Warning: Forcing a time zone like this is not advised, as Docker containers could potentially be launched in many geographic locations. Ideally code should not make assumptions about the local time zone setting of a server.
Bouncing in this context refers to how PaaSTA deploys new version of a service or handles change in its configuration. (soa-config changes: memory, cpu, environment variables, etc).
See the dedicated bounce docs on how PaaSTA does this in a safe (by default) manner.
Draining is the process to stop instances of an old service from receiving traffic. PaaSTA supports pluggable drain methods for service authors to mark services up and down in their environments.
Current master has three draining methods:
- noop - This draining method skips draining completely. Service instances are killed as needed.
- test - This draining method uses class variables to keep track of instances that are marked down to drain and instances that have stopped receiving traffic.
- hacheck - hacheck is
used at Yelp to provide APIs to query and change state of a service instance.
The hacheck draining method requests hacheck to mark down an instance. HAProxy
checks with hacheck periodically to keep its view of instance state up-to-date.
The hacheck draining method will wait for a configurable
delayto make sure HAProxy has the update before considering safe to kill an instance. Note that the hacheck draining method sets an expiration when marking an instance down on hacheck. hacheck will drop the down state if it receives a status query after expiration.
PaaSTA gives you a few Sensu-powered monitoring checks for free:
- setup_marathon_job: Alerts when a Marathon service cannot be deployed or bounced for some reason. It will resolve when a service has been successfully deployed/bounced.
- check_marathon_services_replication: runs periodically and sends an alert if fewer than 50% of the requested instances are deployed on a cluster. If the service is registered in Smartstack it will look in Smartstack to count the available instances. Otherwise it counts the number of healthy tasks in Mesos.
The PaaSTA command line¶
The PaaSTA command line interface,
paasta, gives users of PaaSTA the
ability to inspect the state of services, as well as stop and start existing
services. See the man pages for a description and detail of options for any
individual paasta command. Some of the most frequently used commands are
paasta start- sets the desired state of the service instance to ‘started’. In the case of long-running services, this will mean ensuring that the number of instances of your application matches that set in your soa-configs.
paasta stop- sets the desired state of the service instance to ‘stop’. The result of this for long running tasks is that your tasks are shutdown using whichever drain method you have specified, and tells PaaSTA that the number of instances of your task deployed should be 0. In the case of scheduled tasks, this tells PaaSTA to ensure that your task is no longer scheduled. NB:
paasta stopis a temporary measure; that is, it’s effect only lasts until you deploy a new version of your service. That means that if you run
paasta stopand push a version of the docker image serving your service, then paasta will reset the effect of