Ansible: Verwaltung komplexer Inventories

Ausgangsbasis

Jeder der einmal im Bereich Hosting oder Server-Administration tätig war kam nicht umhin einmal von Ansible gehört zu haben. Um mir den Arbeitsalltag zu erleichtern habe ich bei meinem Arbeitgeber angefangen Ansible für Konfigurations und Installations-Tasks zu verwenden die eine Reihe von Systemen betreffen.

Ich arbeite hier mit einem Inventory das in etwa 75 Server auf zwei IaaS Plattformen und drei nennen wir es Stages umfasst. Zu jedem Kunden gehört eine Reihe von Servern die sich entweder vollständig auf IaaS1 oder IaaS2 befindet. Jeder Kunde besitzt bis zu 3 Stages (Prod, Test, Dev).

Es soll die Möglichkeit gegeben sein auf allen Systemen eines IaaS Aktionen auszuführen. Beispielsweise um die Virtualisierungs-Agents zu aktualisieren oder Auswertungen durchzuführen.
Jedoch soll es auch möglich sein alle Systeme eines Kunden speziell zu behandeln um beispielsweise Zugänge und Benutzer zu verwalten. Es soll aber auch möglich sein alle Systeme einer Stage anzusteuern um beispielsweise Security-Patches einzuspielen.

Bisher wurde das mit einer sehr komplexen und ewig langen Inventory-Datei gelöst.
Der neue Ansatz macht sich eine der Ansible Best-Practices zu nutze.

Inventory Directories

Die Lösung ist die Verwendung von Directories als Inventories.

Dazu konfiguriert man in der ansible.cfg folgendes:

inventory      = /etc/ansible/inventories/

Man hat nun die Möglichkeit eine Struktur nach folgendem Bild zu erstellen.

Der Ordner repräsentiert die Stage, celapp könnte Customer 1 sein, celconversion Customer 2 und so weiter. Zusätzlich befinden sich noch je ein Ordner für Gruppen- und Host-Variablen im Inventory.

hypervtest ist lediglich ein Meta-Inventory welches die Gruppen die in den einzelnen Hostfiles verwendet werden zusammenfügt.

Vorteil ist, dass alle Anwendungsfälle die oben beschrieben wurden sich mit der Verwendung von Pattern in den Playbooks abdecken lassen. Ein weiterer Vorteil ist, dass die Sub-Hostdateien für einzelne Kunden leichter wartbar sind und auch automatisiert generiert werden können.

Nachteil ist eine steigende Komplexität des kompletten Inventories.

%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22%22%20style%3D%22fillColor%3D%23AE4132%3BstrokeColor%3Dnone%3Bopacity%3D30%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22440%22%20y%3D%22180.00000000000003%22%20width%3D%22210%22%20height%3D%2299.99999999999994%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%223%22%20value%3D%22%22%20style%3D%22fillColor%3D%2310739E%3BstrokeColor%3Dnone%3Bopacity%3D30%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%2297%22%20y%3D%22326%22%20width%3D%22863%22%20height%3D%22100%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%224%22%20value%3D%22Prod%22%20style%3D%22rounded%3D1%3BfillColor%3D%2323445D%3BgradientColor%3Dnone%3BstrokeColor%3Dnone%3BfontColor%3D%23FFFFFF%3BfontStyle%3D1%3BfontFamily%3DTahoma%3BfontSize%3D14%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22450%22%20y%3D%22190%22%20width%3D%22190%22%20height%3D%2280%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%225%22%20value%3D%22IaaS%201%22%20style%3D%22rounded%3D1%3BfillColor%3D%2323445D%3BgradientColor%3Dnone%3BstrokeColor%3Dnone%3BfontColor%3D%23FFFFFF%3BfontStyle%3D1%3BfontFamily%3DTahoma%3BfontSize%3D14%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22107%22%20y%3D%22335.5%22%20width%3D%22190%22%20height%3D%2280%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%226%22%20value%3D%22%22%20style%3D%22edgeStyle%3DelbowEdgeStyle%3Belbow%3Dvertical%3BstrokeWidth%3D2%3Brounded%3D0%3BentryX%3D0.5%3BentryY%3D0%3BentryDx%3D0%3BentryDy%3D0%3B%22%20edge%3D%221%22%20source%3D%224%22%20target%3D%227%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22-23%22%20y%3D%22135.5%22%20width%3D%22100%22%20height%3D%22100%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22-23%22%20y%3D%22235.5%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%2277%22%20y%3D%22135.5%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%227%22%20value%3D%22IaaS%202%22%20style%3D%22rounded%3D1%3BfillColor%3D%2323445D%3BgradientColor%3Dnone%3BstrokeColor%3Dnone%3BfontColor%3D%23FFFFFF%3BfontStyle%3D1%3BfontFamily%3DTahoma%3BfontSize%3D14%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22757%22%20y%3D%22335.5%22%20width%3D%22190%22%20height%3D%2280%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%228%22%20value%3D%22%22%20style%3D%22edgeStyle%3DelbowEdgeStyle%3Belbow%3Dvertical%3BstrokeWidth%3D2%3Brounded%3D0%3BentryX%3D0.5%3BentryY%3D0%3BentryDx%3D0%3BentryDy%3D0%3BexitX%3D0.5%3BexitY%3D1%3BexitDx%3D0%3BexitDy%3D0%3B%22%20edge%3D%221%22%20source%3D%224%22%20target%3D%225%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22-13%22%20y%3D%22145.5%22%20width%3D%22100%22%20height%3D%22100%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22862%22%20y%3D%22275.5%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22862%22%20y%3D%22346%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%229%22%20value%3D%22%22%20style%3D%22fillColor%3D%23ffe6cc%3BstrokeColor%3D%23d79b00%3Bopacity%3D30%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%2297%22%20y%3D%22480%22%20width%3D%22423%22%20height%3D%22100%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2210%22%20value%3D%22%26lt%3Bb%26gt%3B%26lt%3Bfont%20style%3D%26quot%3Bfont-size%3A%2017px%26quot%3B%26gt%3BCustomer%201%26lt%3B%2Ffont%26gt%3B%26lt%3B%2Fb%26gt%3B%22%20style%3D%22rounded%3D1%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3BfillColor%3D%23ffe6cc%3BstrokeColor%3D%23d79b00%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22107%22%20y%3D%22490%22%20width%3D%22193%22%20height%3D%2280%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2211%22%20value%3D%22%26lt%3Bb%26gt%3B%26lt%3Bfont%20style%3D%26quot%3Bfont-size%3A%2017px%26quot%3B%26gt%3BCustomer%203%26lt%3B%2Ffont%26gt%3B%26lt%3B%2Fb%26gt%3B%22%20style%3D%22rounded%3D1%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3BfillColor%3D%23ffe6cc%3BstrokeColor%3D%23d79b00%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22320%22%20y%3D%22490%22%20width%3D%22193%22%20height%3D%2280%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2212%22%20value%3D%22%22%20style%3D%22fillColor%3D%23dae8fc%3BstrokeColor%3D%236c8ebf%3Bopacity%3D30%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22537%22%20y%3D%22480%22%20width%3D%22423%22%20height%3D%22100%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2213%22%20value%3D%22%26lt%3Bb%26gt%3B%26lt%3Bfont%20style%3D%26quot%3Bfont-size%3A%2017px%26quot%3B%26gt%3BCustomer%202%26lt%3B%2Ffont%26gt%3B%26lt%3B%2Fb%26gt%3B%22%20style%3D%22rounded%3D1%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3BfillColor%3D%23ffe6cc%3BstrokeColor%3D%23d79b00%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22755.5%22%20y%3D%22490%22%20width%3D%22193%22%20height%3D%2280%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2214%22%20value%3D%22%26lt%3Bb%26gt%3B%26lt%3Bfont%20style%3D%26quot%3Bfont-size%3A%2017px%26quot%3B%26gt%3BCustomer%204%26lt%3B%2Ffont%26gt%3B%26lt%3B%2Fb%26gt%3B%22%20style%3D%22rounded%3D1%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3BfillColor%3D%23ffe6cc%3BstrokeColor%3D%23d79b00%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22545%22%20y%3D%22490%22%20width%3D%22193%22%20height%3D%2280%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2215%22%20value%3D%22%22%20style%3D%22edgeStyle%3DelbowEdgeStyle%3Belbow%3Dvertical%3BstrokeWidth%3D2%3Brounded%3D0%3BexitX%3D0.5%3BexitY%3D1%3BexitDx%3D0%3BexitDy%3D0%3B%22%20edge%3D%221%22%20source%3D%225%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%2215%22%20y%3D%22505.5%22%20width%3D%22100%22%20height%3D%22100%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22572.8571428571429%22%20y%3D%22630.1428571428571%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22202%22%20y%3D%22490%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2216%22%20value%3D%22%22%20style%3D%22edgeStyle%3DelbowEdgeStyle%3Belbow%3Dvertical%3BstrokeWidth%3D2%3Brounded%3D0%3BexitX%3D0.5%3BexitY%3D1%3BexitDx%3D0%3BexitDy%3D0%3BentryX%3D0.5%3BentryY%3D0%3BentryDx%3D0%3BentryDy%3D0%3B%22%20edge%3D%221%22%20source%3D%227%22%20target%3D%2213%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22663%22%20y%3D%22510%22%20width%3D%22100%22%20height%3D%22100%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22850%22%20y%3D%22419.78571428571433%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22850%22%20y%3D%22494.5%22%20as%3D%22targetPoint%22%2F%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2217%22%20value%3D%22Server%201%22%20style%3D%22rounded%3D0%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22140%22%20y%3D%22600%22%20width%3D%22120%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2218%22%20value%3D%22Server%202%22%20style%3D%22rounded%3D0%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22140%22%20y%3D%22680%22%20width%3D%22120%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2219%22%20value%3D%22Server%201%22%20style%3D%22rounded%3D0%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22356.5%22%20y%3D%22600%22%20width%3D%22120%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2220%22%20value%3D%22Server%202%22%20style%3D%22rounded%3D0%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22356.5%22%20y%3D%22680%22%20width%3D%22120%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2221%22%20value%3D%22Server%201%22%20style%3D%22rounded%3D0%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22580%22%20y%3D%22600%22%20width%3D%22120%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2222%22%20value%3D%22Server%202%22%20style%3D%22rounded%3D0%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22580%22%20y%3D%22680%22%20width%3D%22120%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2223%22%20value%3D%22Server%201%22%20style%3D%22rounded%3D0%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22792%22%20y%3D%22600%22%20width%3D%22120%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2224%22%20value%3D%22Server%202%22%20style%3D%22rounded%3D0%3BwhiteSpace%3Dwrap%3Bhtml%3D1%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22792%22%20y%3D%22680%22%20width%3D%22120%22%20height%3D%2260%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2225%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3BentryDx%3D0%3BentryDy%3D0%3BexitX%3D0.5%3BexitY%3D1%3BexitDx%3D0%3BexitDy%3D0%3B%22%20edge%3D%221%22%20source%3D%2210%22%20target%3D%2217%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%2240%22%20y%3D%22630%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22100%22%20y%3D%22620%22%20as%3D%22targetPoint%22%2F%3E%3CArray%20as%3D%22points%22%3E%3CmxPoint%20x%3D%22203%22%20y%3D%22590%22%2F%3E%3CmxPoint%20x%3D%22110%22%20y%3D%22590%22%2F%3E%3CmxPoint%20x%3D%22110%22%20y%3D%22630%22%2F%3E%3C%2FArray%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2226%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3BentryDx%3D0%3BentryDy%3D0%3BexitX%3D0.5%3BexitY%3D1%3BexitDx%3D0%3BexitDy%3D0%3B%22%20edge%3D%221%22%20source%3D%2210%22%20target%3D%2218%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22213.20689655172418%22%20y%3D%22580.3103448275863%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22149.75862068965512%22%20y%3D%22640.3103448275863%22%20as%3D%22targetPoint%22%2F%3E%3CArray%20as%3D%22points%22%3E%3CmxPoint%20x%3D%22203%22%20y%3D%22590%22%2F%3E%3CmxPoint%20x%3D%22160%22%20y%3D%22590%22%2F%3E%3CmxPoint%20x%3D%22110%22%20y%3D%22590%22%2F%3E%3CmxPoint%20x%3D%22110%22%20y%3D%22710%22%2F%3E%3C%2FArray%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2227%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3BentryDx%3D0%3BentryDy%3D0%3BexitX%3D0.5%3BexitY%3D1%3BexitDx%3D0%3BexitDy%3D0%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22416.70689655172396%22%20y%3D%22569.8103448275863%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22353.2586206896551%22%20y%3D%22709.8103448275863%22%20as%3D%22targetPoint%22%2F%3E%3CArray%20as%3D%22points%22%3E%3CmxPoint%20x%3D%22416.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22373.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22323.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22323%22%20y%3D%22660%22%2F%3E%3CmxPoint%20x%3D%22323.5%22%20y%3D%22709.5%22%2F%3E%3C%2FArray%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2228%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3BentryDx%3D0%3BentryDy%3D0%3BexitX%3D0.5%3BexitY%3D1%3BexitDx%3D0%3BexitDy%3D0%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22416.70689655172396%22%20y%3D%22569.8103448275863%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22353.2586206896551%22%20y%3D%22629.8103448275863%22%20as%3D%22targetPoint%22%2F%3E%3CArray%20as%3D%22points%22%3E%3CmxPoint%20x%3D%22416.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22323.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22323.5%22%20y%3D%22629.5%22%2F%3E%3C%2FArray%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2229%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3BentryDx%3D0%3BentryDy%3D0%3BexitX%3D0.5%3BexitY%3D1%3BexitDx%3D0%3BexitDy%3D0%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22641.7068965517242%22%20y%3D%22569.8103448275863%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22578.2586206896551%22%20y%3D%22629.8103448275863%22%20as%3D%22targetPoint%22%2F%3E%3CArray%20as%3D%22points%22%3E%3CmxPoint%20x%3D%22641.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22548.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22548.5%22%20y%3D%22629.5%22%2F%3E%3C%2FArray%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2230%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3BentryDx%3D0%3BentryDy%3D0%3BexitX%3D0.5%3BexitY%3D1%3BexitDx%3D0%3BexitDy%3D0%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22853.2068965517242%22%20y%3D%22569.8103448275863%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22789.7586206896549%22%20y%3D%22629.8103448275863%22%20as%3D%22targetPoint%22%2F%3E%3CArray%20as%3D%22points%22%3E%3CmxPoint%20x%3D%22853%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22760%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22760%22%20y%3D%22629.5%22%2F%3E%3C%2FArray%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2231%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3BentryDx%3D0%3BentryDy%3D0%3BexitX%3D0.5%3BexitY%3D1%3BexitDx%3D0%3BexitDy%3D0%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22641.7068965517242%22%20y%3D%22569.8103448275863%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22578.2586206896551%22%20y%3D%22709.8103448275863%22%20as%3D%22targetPoint%22%2F%3E%3CArray%20as%3D%22points%22%3E%3CmxPoint%20x%3D%22641.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22598.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22548.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22548%22%20y%3D%22660%22%2F%3E%3CmxPoint%20x%3D%22548.5%22%20y%3D%22709.5%22%2F%3E%3C%2FArray%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3CmxCell%20id%3D%2232%22%20value%3D%22%22%20style%3D%22endArrow%3Dclassic%3Bhtml%3D1%3BentryX%3D0%3BentryY%3D0.5%3BentryDx%3D0%3BentryDy%3D0%3BexitX%3D0.5%3BexitY%3D1%3BexitDx%3D0%3BexitDy%3D0%3B%22%20edge%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20width%3D%2250%22%20height%3D%2250%22%20relative%3D%221%22%20as%3D%22geometry%22%3E%3CmxPoint%20x%3D%22853.7068965517237%22%20y%3D%22569.8103448275863%22%20as%3D%22sourcePoint%22%2F%3E%3CmxPoint%20x%3D%22790.2586206896549%22%20y%3D%22709.8103448275863%22%20as%3D%22targetPoint%22%2F%3E%3CArray%20as%3D%22points%22%3E%3CmxPoint%20x%3D%22853.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22810.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22760.5%22%20y%3D%22589.5%22%2F%3E%3CmxPoint%20x%3D%22760%22%20y%3D%22660%22%2F%3E%3CmxPoint%20x%3D%22760.5%22%20y%3D%22709.5%22%2F%3E%3C%2FArray%3E%3C%2FmxGeometry%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E