Technical Proposal 1. About ETA2U / ITPS ETA2U is VMware Solution Provider certified at Enterprise level, starting with 2008, and has rolled-out more than 50 VMware projects for various customers, involving hundreds of ESX hosts and thousands of VM’s. More than half of there environments are currently under support contracts. ETA2U is Platinum Level Partner – Oracle Database Specialization. Also ETA2U is Oracle Application Partner (Enterprise Business Suite), and the only company certified as Oracle Business Accelerators in Romania. ETA2U is Oracle Value Added Distributor (one of the only 2 VAD’s in Romania).
2. Project Team The services described within this document will be fulfilled by a staff including minimum the following resources: -
VMware Certified Professional (VCP) x 2
-
Oracle 11i System Administration Certified Expert
-
Oracle Database 10g Administrator Certified Professional
-
Oracle Database 10g Administrator Certified Associate
-
HelpDesk personnel for registering ticketing, follow-up and escalation
-
Technical Account Manager for RCA Analysis, reporting and assessment
-
On Site 1st level engineer for immediate on site interventions (localized in Oradea)
-
Account Manager for incident escalation/customer feedback
3. Serviced INCLUDED in the monthly fee: Monitoring services for proactive remediation of problems, trend analysis for systems HelpDesk system operation for registering, follow-up on incidents Break/Fix remedy for all incidents related to VMware infrastructure (ESX hosts, hypervisors, SAN Switches/HBA’s, Storage Units, vCenter configurations, backup integrations, networking integrations, etc.) Consulting services for optimizing current infrastructure Recurrent maintenance tasks for systems (patching, upgrades, alerts)
ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
Administering backup processes for existing virtual environment, using Tivoli Storage Manager (licenses must be supplied by Celestica). Service scope includes Vmware platform integration and Tape Library, backup writing process, including inventory, mounting, restore operations Monthly reporting from monitoring tool (including uptime per services/systems/parameters), HelpDesk tool, and other specific applications (VMware Health Check). Monthly reporting may include RCA (Root Cause Analysis) upon request by Celestica. Remote access to the monitoring tool for Celestica personnel, providing Real-time information on health status. Service category VMware
Operation details o
monthly report containing incidents / "VMWARE HEALTH CHECK" report
o
Break/Fix incident execution for all issues related to VMware environment components, within defined SLA
o
if needed, escalating VMware incidents, using Celestica support codes.
o
Optimizing and monitoring parameters for disk subsystem, network, memory and CPU for VMware Environment
o
Updates and install for VMware Tools in guest OS
o
patch management
o
storage administration (connectivity, multipath, load balancing, etc.)
o
Root Cause Analysis for specific incidents
o
Supervising and optimizing backup execution (Autoloader troubleshooting, fine tuning backup plan, consistency validation for backup jobs, etc)
Oracle OS
o
OS install and configure
o
Driver optimization, performance for network/disk/memory/CPU
o
Monitoring system parameters
o
Patching and integrity assurance for operating system
o
System upgrades towards newer Oracle versions1
Oracle DBA –
o
Remote Fix operations for DB-related problems
services included in
o
Escalating bug-type issues to Oracle (depending upon the active
the monthly fee
Oracle support parameters) and joint operations with Oracle tech
1
Price for these services is not included in the specified monthly fee, and will be considered additional work, quoted when needed. Upgrade/Migration methodology will be published by Oracle. ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
personnel for implementing changes/remediation o
real-time db monitoring (uptime, tablespace, usage, indexes)
o
backup process monitoring for databases (using Dataguard, TSM, Local)
o
monthly report with Oracle incidents
o
restarting oracle instances (in case of audits, for example)
o
running various existing scripts (PLSQL, UTL_MAIL, etc.)
o
consulting on operations performed by Celestica personnel (table / indexes relocation, tablespaces modifications, etc.)
Oracle DBA –
o
new Oracle server installation
services at extra
o
upgrading to new database versions
charge
o
migrating databases from one server to another
o
other types of operations not included in the above section
ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
4. In-Scope components In-Scope equipment and systems are described in the following table.
Environment Type Vmware Infrastructure Server Server Server Storage Oradea, RO Storage Server Server Storage Storage Server Frankenburg, Server AT Server Storage
Role
Vendor Model
ESX 4.1 ESX 4.1 ESX 4.1
IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM IBM
x3650 M2 x3650 M2 x3650 M2 DS3400 ESM3400 x3650 M2 x3650 M2 DS3400 ESM3400 X3650 M3 X3650 M3 X3650 M3 DS3512
ESX 4.1 ESX 4.1
ESX 4.1 ESX 4.1 ESX 4.1
Name
HA configured ORAVST01
To be integrated in existing cluster system Frankenburg VMware Infrastructure
ORASBD01
BD active de productie
Cluster
ORAODB02 ORASBD02 ORAODA01
BD active de productie
Cluster
Server
Database (Linux + Oracle)
IBM
x3650 M2
ORAODB01
Server
Database (Linux + Oracle)
IBM
x3650 M2
Server Server Server
Database (Linux + Oracle) Database (Linux + Oracle) Database (Linux + Oracle)
IBM IBM IBM
x3650 M3 x3650 M3 x3650 M3
TSM
Server Server Library Library
Backup (IBM TSM) Backup (IBM TSM)
IBM IBM SUN SUN
x3550 x3550 StorageTek SL500 StorageTek SL500
ODC AUSTRIA
Server
Database (Linux + Oracle)
IBM
x3650 M2
ATFBMS06
Server
Database (Linux + Oracle)
IBM
x3650 M2
ATFBMS07
ODC
ODC RIM
Observations
BD active de productie
5. Service Level Agreement(SLA) ETA2U/ETA2U, through it’s dedicated support team, shall operate a HelpDesk system acknowledging and registering support calls via phone 0256.277.577 or mail:
[email protected]. All registered incidents will be operated according to the “Platinum” SLA defined below: Platinum SLA :
ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
Priority classification Severity 4 - Urgent Problems affecting completely/seriously Celestica’s activity. Critical systems are down or otherwise unusable.
Severity 3 - High Problems affecting specific departments, or moderately impacting Celestica’s activity. Critical systems are partially available/response time is slower, but operations can continue.
Acknowledge target
Fix target2
Service availability
30 min
2h
2h
4h
1h
4h
24x7 business hours 24x7 non business hours 24x7 business hours
2h
8h
24x7 non business hours
2h
8h
8x5
8h
16h
8x5
Minimum
Target
99,8%
99,9%
99,8%
99,9%
Severity 2 – Medium Problems affecting a single user, or low impact for Celestica’s activity. Critical systems are fully available, performance may be slightly degraded.
Severity 1 – Low Request not classified as incidents. For new feature deployment, fine tuning, performance optimizations, etc.
Service availability VMware Cluster (ESX, Storage, Network connectivity, backup)
Oracle Infrastructure (Linux OS, DB)
Notes: • Service availability is determined annually for the first year, and then monthly, but reporting downtime to the previously 12 calendar months. • Fix target time will be excepted from SLA definitions in the following cases: o Problems are generated from other systems than the ones InScope of current contract (ex: power outages, virus infections, data loss/data theft, other networking malfunctions, Active Directory/DNS/DHCP problems, etc.) o Hardware problems generated by existing IT equipment. Fix Time will be assumed once Celestica is in possession of adequate spare components. o Problems related to "Force Majeure" • INCIDENT = Unplanned interruption for an IT Service, or quality falling bellow “normal” functioning levels. • Fix Time will be assumed once a request becomes registered within HelpDesk system • Service Provider can be penalized for not meeting SLA or other contract-specific engagements with the following schematic: 5% off monthly contract value, for each hour delay over the Fix Time designated 1% off annual contract value for each 0,1 percentage falling below minimum service availability
2
Fix target time may be altered by recovery procedures publishe by Oracle or VMware, and by the size of the data in tranzit. The duration for data copying/restoring cannot be considered Fix target time violation, as long as these processes are required. ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
6. Financial Proposal
9960 USD + VAT / month for the services defined above Notes: • For VMware infrastructure the cost is fixed, and includes unlimited number of incidents, covering the existing infrastructure detailed in Chapter 4. • Oracle DBA Services cover 3 primary databases, 3 standby databases and 1 archive database. Active Support Contract from Oracle must be in place and valid, covering Oracle database licenses, and ETA2U personnel must be assigned required CSI for the duration of this contract. For incident solving, ETA2U will follow Oracle procedures for each specific incident. • Celestica will be awarded a monthly credit consisting of 24 consulting hours, which can be used for extra charge services, without paying additional fees. Customer shall be billed for any additional services hours above 24 hrs/month at a rate of 75 USD/h during Business Hours, and 150 USD/h during Off Business Hours. Any unused credit during one month will not be shifted over to the next month. • Support for Frankenburg location will be provided either remote, or using “remote hands” from local Celestica personnel. For advanced onsite procedures, additional cost for Transport & Accomodation will be charged. • New features, technology upgrades/refresh, major design changes, restructuring critical services with changes in topology will be considered additional services, and will not be covered by existing monthly prices. Each of these will be treated individually, and a separate price quote will be issues prior to service engagement. Services will be executed pending Celestica’s authorization for the additional cost. • If the hardware/software equipment/infrastructure will be extended or shortened, we will reserve the right to modify the prices accordingly.
Cristian Leoveanu
Business Unit Manager
[email protected]
Document # / date: 271/23.11.2012 Content : support services VMware/Oracle Customer name: Celestica Romania
Office: +40 356-452.051 Fax: +40 356-452.052 Mobile: +40 744 531 401
ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
7. Specific tools used for support contract 7.1.
• • • • • • •
•
HelpDesk
Centralized Help Desk tool, allows for complex management and reporting for incidents, including ITIL-compliant system for registering and managing incidents. Real Time response for opened tickets 3-level vertical structure for support teams, to insure proper prioritizing for incidents according to severity classification Horizontal type-specific organization for support teams, each with dedicated knowledge for a specific environment Complex prioritization method, based on tasks and cases Configuration Management Database (CMDB) covering existing systems. Activity reports are available based on number of incidents, response time, Fix Time, Assets, User requests, etc. These reports can span specific time intervals, or be generic – weekly, monthly, yearly. IT hardware equipment tagging, allowing for user/department assigning.
Samples of monthly-generated reports you can find below. Timeline evolution and classification can also be used in determining service-level trends, aside from the regular Feedback Forms.
Report cases / severity Count of Case Number Created On Sep Oct Nov Total
Severity S1-Affects Business
1 1
S2- Affects Department/Group 2 1 2 5
S3-Affects User
Total
20 14 9 43
22 15 12 49
ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
Report cases/problem type Count of Case Number Case Type Problem Question Request Total
Total 44 11 2 57
The ticketing system is based on a Microsoft CRM backend system. Below you can see some report screenshot
ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
7.2.
Monitoring tool for VMware infrastructure
ETA2U/ETA2U will provide a online, cloud-based monitoring tool, capable of interacting with all of the components of the ecosystem. The NAGIOS-based system has the following features and capabilities • Equipment status monitoring (UP/DOWN) • Services status monitoring, with individual services in specific hosts (SMTP, HTTP, FTP, etc.) ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
• • • • • • • • •
•
•
Internal Server/OS parameters monitoring (free disk space, memory load, CPU load, number of running processes) Grouping specific resources in a group pool, based on component interactions. Automatic map generation with host/devices layout Automatic uptime reporting (individual hosts, specific services or groups) Detailed services reporting, Critical/Warning threshold definition Granular alerting (based on groups, destination, threshold, etc.) History events for a specific device Variable threshold alerting (email, SMS, SNMP trap) Single pane of glass for multiple environments, hardware and software (Windows, Linux, Solaris, servers, Cisco, Juniper, Checkpoint, firewalls and routers, UPSes with management capabilities, SAN-device equipments, etc. Multiple collection capabilities – active and passive, using bridgeheads within the monitored network or simply by telnet-in specific ports from outside the corporate network. Graphing capabilities with interactive charts
We are also attaching several application screenshots.
ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
ETA2U - str. Gh. Dima nr. 1 - tel. 0356.452.051 - fax. 0356.452.052 -
[email protected] - www.ETA2U.ro
7.3.
ORACLE Monitoring & Support:
For monitoring purposes, we will be using a couple of scripts which will provide visibility and proactive support for databases and operating system through notifications and events such as: • Space gained in tablespaces • Invalid objects in databases • Free disk space • High growth segments within last 24 hrs • Stalled processes or processes using high amount of CPU time Tools used for administrative operations with databases can be Toad or Oracle Enterprise Manager, depending upon licenses provided by Celestica.
Email Output for monitoring OS and DB (in addition to Nagios sensors) DB Node informations
almost full tablespaces Tablespace Allocated_Space Free_Space Used Max_size De_Alocat Name MB MB % MB MB ------------------------------ --------------- ---------- ---------- ---------- ------------------------------
Nr_invalid_objects ------------------15
Disk Space: Filesystem /dev/sda1 tmpfs /dev/sdc1 /dev/sdb1
Size 128G 16G 71G 275G
Used 98G 0 612G 165G
Avail 24G 16G 52G 96G
Use% 81% 0% 93% 64%
Mounted on / /dev/shm /d01 /d02
Less than 10% of available extents in segments (maxextents > 1) segment_type owner segment_nameformat extents max_extents --------------- ---------- -------------------------------- ---------- -----------
Segments with abnormal growth within last 24 hrs (3 times) Segment_name Segment_ty Owner Extents Max_Extents Crestere ------------------------------ ---------- ---------- ---------- ----------- ---------OE_PRICE_ADJ_ATTRIBS TABLE ONT 11353 2147483645 6 MTL_TRANSACTION_ACCOUNTS TABLE INV 29890 2147483645 14 MTL_MATERIAL_TRANSACTIONS_N9 INDEX INV 8196 2147483645 3 MTL_CST_ACTUAL_COST_DETAILS_U1 INDEX INV 8001 2147483645 4 WRH$_SQLSTAT TABLE PART SYS 12 2147483645 5 ITION
RA_CUST_TRX_LINE_GL_DIST_ALL TABLE AR OE_PRICE_ADJUSTMENTS TABLE ONT MTL_TRANSACTION_ACCOUNTS_N5 INDEX INV MTL_MATERIAL_TRANSACTIONS TABLE INV WSH_DELIVERY_DETAILS TABLE WSH RA_CUSTOMER_TRX_LINES_ALL TABLE AR MTL_MATERIAL_TRANSACTIONS_N3 INDEX INV MTL_TRANSACTION_ACCOUNTS_N3 INDEX INV MTL_MATERIAL_TRANSACTIONS_N7 INDEX INV OE_PRICE_ADJ_ATTRIBS_N2 INDEX ONT OE_ORDER_LINES_ALL TABLE ONT MTL_TXN_REQUEST_LINES TABLE INV CST_INV_LAYERS TABLE BOM MTL_MATERIAL_TRANSACTIONS_N8 INDEX INV MTL_MATERIAL_TRANSACTIONS_N15 INDEX INV MTL_CST_LAYER_ACT_CST_DTLS_U1 INDEX INV MTL_MATERIAL_TRANSACTIONS_N5 INDEX INV WRH$_LATCH_MISSES_SUMMARY TABLE PART SYS ITION WRH$_SYSSTAT
TABLE PART SYS
12346 2147483645 13340 2147483645 9823 2147483645 55930 2147483645 21741 2147483645 31231 2147483645 8678 2147483645 7885 2147483645 5551 2147483645 6642 2147483645 41024 2147483645 9152 2147483645 6578 2147483645 7669 2147483645 8030 2147483645 9155 2147483645 6695 2147483645 5 2147483645
5 8 5 25 13 12 3 4 3 3 24 5 3 3 4 5 4 3
4 2147483645
3
17700 2147483645 19355 2147483645 9306 2147483645 6740 2147483645
8 9 5 3
ITION MTL_CST_LAYER_ACT_COST_DETAILS TABLE INV MTL_CST_ACTUAL_COST_DETAILS TABLE INV MTL_TRANSACTION_ACCOUNTS_N2 INDEX INV MTL_MATERIAL_TRANSACTIONS_N1 INDEX INV
MTL_TRANSACTION_ACCOUNTS_N6 INDEX OE_ORDER_LINES_N11 INDEX ONT MTL_TRANSACTION_ACCOUNTS_N1 INDEX WRH$_PARAMETER TABLE PART SYS ITION
INV INV
RCV_TRANSACTIONS TABLE PO MTL_MATERIAL_TRANSACTIONS_U2 INDEX INV MTL_TRANSACTION_ACCOUNTS_N7 INDEX INV AR_TRANSACTIONS_REP_ITF TABLE AR WRH$_LATCH TABLE PART SYS ITION WRH$_LATCH_MISSES_SUMMARY_PK INDEX PART SYS ITION
8490 2147483645 5170 2147483645 7116 2147483645 4 2147483645
1981 2147483645 3 7427 2147483645 4 10671 2147483645 4 14332 2147483645 13 7 2147483645
5 2147483645
Processes using more than 50% CPU Pid Time Program_Name CPU User --------------- --------------- --------------- ---------- --------------10031 00:00:01 oracle 96.5 oraprod
Swap Usage: Mem:
3 3 4 3
total used free shared buffers cached 33005976 23041812 9964164 0 1894084 839020
3
-/+ buffers/cache: 20308708 12697268 Swap: 4192956 0 4192956
alert.log messages: 286302,286304d286301 < 1545324:ALTER SYSTEM ARCHIVE LOG < 1545334:ALTER SYSTEM ARCHIVE LOG < 1545348:ALTER SYSTEM ARCHIVE LOG
TOAD screenshots:
