A Multicriteria Decision Support System for Competence-Driven Project Portfolio Selection

Electronic version of an article published in International Journal of Information Technology & Decision Making, vol. 8, no. 2, pp. 379–401, 2009; DOI: 10.1142/S0219622009003429 c

World Scientific Publishing Company, http://www.worldscinet.com/ijitdm/ijitdm.shtml

A MULTICRITERIA DECISION SUPPORT SYSTEM FOR COMPETENCE-DRIVEN PROJECT PORTFOLIO SELECTION∗

CHRISTIAN STUMMER† and ELMAR KIESLING‡ Department of Business Administration, University of Vienna, Bruenner Str. 72, A-1210 Vienna, Austria † Corresponding author: [email protected], Tel.: +431 4277-38146; fax: -38144 ‡ [email protected] WALTER J. GUTJAHR Department of Statistics and Decision Support Systems, University of Vienna, Universitaetsstr. 5/3, A-1010 Vienna, Austria [email protected]

The systematic and proactive development of human resources is of major importance in organizations that rely heavily on the competencies of their employees when engaging in innovative endeavors. Human capital, however, is not only a resource required for conducting research, but also the eventual result of that research. When selecting a research portfolio, the decision-maker thus needs to take into consideration both current and future competence requirements, as well as other financial and non-financial objectives and constraints. We introduce a proper multicriteria decision support system (MCDSS) that first determines the set of Pareto-efficient solutions and then allows the decision-maker to interactively filter and/or explore this set in various ways. Its practical application is demonstrated by means of a showcase at the Electronic Commerce Competence Center (EC3) in Vienna, Austria. Keywords: Multicriteria decision support system; Interactive decision analysis; Project portfolio selection; Competence development; Visual interfaces; Interactive navigation

1. Introduction Technology-intensive companies or research-focused institutions must be able to identify promising new projects at an early stage so that the necessary resources can be allocated to these activities. As these organizations more often than not rely on the competencies of their employees, they typically are not first and foremost concerned with the financial resources to be distributed amongst a set of research and development (R&D) project opportunities, but rather with the allocation of human ∗ Supported

by the Austrian Science Fund (Grant No. L264-N13). 1

2

C. Stummer, E. Kiesling & W. J. Gutjahr

capital. The reason for this lies in the fact that available expertise (i. e., competence) primarily determines whether a research or innovation endeavor is successful or if it is doomed to fail because of a (not appropriately anticipated) lack of critical intellectual capabilities of some sort. The project portfolio selection at hand thus also includes the staff assignment1, 2 as well as the proper scheduling of project proposals2–6 , thus creating a challenging decision-making task. This task becomes all the more intriguing due to the dual nature of human capital, which is both a resource indispensable for conducting research and innovation, as well as the eventual result of these activities. Therefore, the problem is characterized by incommensurate and conflicting objectives, such as financial criteria (e. g., costs), performance and, particularly, the competence levels achieved in a number of competence categories (or combinations thereof). Methods from the field of multicriteria decision-making come into play, as a unique optimal decision does not typically exist, but numerous decisions may be suitable. Many early portfolio selection models were based on an optimization approach.7 Given a number of projects and a pool of resources, the portfolio was optimized by converting its projects’ attributes into a single (utility) value. However, the reality of decision aiding is that the client quite often does not have a very clear idea of the problem, at least not clear enough to allow the identification of a model of rationality.8 For this reason, modern multicriteria decision support systems (MCDSS) aim at facilitating decision-makers (DMs) in a process of learning about an issue and about their own and other stakeholders’ perspectives on and preferences relating to that issue. To this end, visual interactive displays are acknowledged to be the most powerful means of communications for the majority of people.9 In this paper, we introduce a new MCDSS that was designed for competencedriven project portfolio selection and applied at the Electronic Commerce Competence Center (EC3) in Vienna, Austria. A two-step procedure is used to obtain a preferred solution: in the first phase, the DM specifies relevant monetary and non-monetary criteria (e. g., expected value for EC3 members, acquired third-party funding or competence levels in crucial competence categories). The MCDSS then determines the set of (Pareto-) efficient portfolio solutions or at least an approximation thereof. In the second phase, the system supports the DM in exploring the solution space until he/she finds the portfolio that provides the individually “best” (i. e., most attractive) mix of objective values. To this end, we have implemented several interactive graphical user interfaces that provide insights from different points of view and/or to different levels of detail. The remainder of this paper is organized as follows: Section 2 provides a formal description of the underlying multicriteria portfolio selection, scheduling and staffing with learning (“MPSSSL”) problem. In Section 3, we introduce the interactive MCDSS while Section 4 illustrates its application in a real-world setting. The paper concludes in Section 5 with a summary and an outlook on further research.

A MCDSS for competence-driven project portfolio selection

3

2. Competence-driven Project Portfolio Selection 2.1. Problem formulation The problem encompasses a project portfolio selection decision on an upper decision level and a project scheduling and staff assignment decision on a lower level. The upper-level decision consists in the choice of a subset of projects, taken from a given set of candidates i = 1, . . . , n. For the following description (formal details can be found in Refs. 10, 11), let the binary indicator variable yi take the value 1 if project i is to be selected, and the value 0 otherwise. Let y = (y1 , . . . , yn ). Projects can be partitioned into tasks, which are indexed by k = 1, . . . , K; we assume that each task k belongs to exactly one project i, whereas a project i can consist of one or several tasks. Further, each task k has an assigned ready time and an assigned due date. The decision has to be made in consideration of certain required human competencies r = 1, . . . , R. We call that part of a task k that requires a particular competency r the work package with index (k, r). It is assumed that task k requires a (so-called “ideal”) work time of dkr in competency r, where dkr is given in advance. We consider a fixed team of employees j = 1, . . . , m. For each employee j, it is supposed that his/her efficiency in each competency r can be quantified as a value γjr , measuring the fraction of ideal work in competency r that he/she is able to deliver within given time, compared to the work of an employee with a “perfect” ability in the respective competency r. If employee j with efficiency γjr works for xjkr time units in work package (k, r), he/she reduces the ideal work time required for work package (k, r) by the amount γjr · xjkr . On the lower decision level, given a decision on a project portfolio, the workload corresponding to the single work packages must be assigned to the staff over time. A fixed planning interval consisting of T periods is considered. Periods are indexed by t = 1, . . . , T . (Typically, a period consists of one month.) Period t starts at time t − 1 and ends at time t. Decision variable xkjrt denotes the amount of real work time in work package (k, r) assigned to employee j in period t. Since efficiencies will be considered as variable over time, we write γjrt instead of γjr in the following, indicating the dependency on time t. The degree to which an employee j possesses a certain competency r at time t is quantified by a real value zjrt , which we call the competence score. We assume that by learning, zjrt increases when employee j works in a project requiring competency r, and that zjrt diminishes due to the so-called knowledge depreciation effect when employee j is not active in competency r. Initial values zjr1 of the competence scores in period 1 are assumed as given. More specifically, we assume: • The efficiency values γjrt depend on the competence scores zjrt via an increasing function ϕ mapping the real line into the interval [0, 1]: γjrt = ϕ(zjrt ). Following literature on organizational learning12, 13 , we assume that ϕ can be modeled by a logistic function with suitably calibrated parameters.

4

C. Stummer, E. Kiesling & W. J. Gutjahr

• The competence score zjrs of employee j in competency r increases from time s = t − 1 to time s = t by the amount ηr

K X

xkjrs − βr ,

(2.1)

k=1

where ηr ≥ 0 is a learning rate, and βr ≥ 0 is a knowledge depreciation rate. The free capacity of employee j in period t is denoted by ajt , expressed in working time units, where we take the normal working time in one period as a working time unit (j = 1, . . . , m; t = 1, . . . , T ). For the definition of the objective functions, suppose that for π = 1, . . . , p, (π) (π) vectors w(π) = (w1 , . . . , wn ) of economic benefits from projects 1 to n are given. (π) The π-th economic benefit value wi can refer to any economic measure of gain as profit, turnover, market share or other. Further, suppose that for κ = 1, . . . , q, (κ) (κ) vectors v (κ) = (v1 , . . . , vR ) of strategic weights of competencies 1 to R are given. (κ) The κ-th strategic weight vr quantifies the importance of competency r in view of long-term strategic considerations from viewpoint κ. For the arrays x = (xkjrt ) and y = (yi ) of decision variables, we define two sets of objective functions as follows: f (π) (y) =

n X

(π)

wi yi

(π = 1, . . . , p)

(2.2)

i=1

and g (κ) (x) =

R X r=1

vr(κ)

m X

∆γjr

(κ = 1, . . . , q).

(2.3)

j=1

Therein, ∆γjr denotes the increment of the efficiency of employee j in competency r between the beginning and the end of the planning interval, i. e., the value γj,r,T +1 − γjr1 . The objective function f (π) (y) in the first set (2.2) measures the (π) (π) economic benefit in a perspective where the values w1 , . . . , wn are assigned to the projects 1, . . . , n. Observe that the value assigned to a project can express different quantities, e. g. profit or turnover, thus providing different evaluations of portfolios and therefore different economic objective functions, which can be analyzed simultaneously. The second set (2.3) of objective functions represents the strategic benefits obtained from the increments of the efficiencies γjrt over the planning horizon. The objective function g (κ) (x) in this set measures the strategic benefit (κ) (κ) obtained if the single competencies are weighed by using the weights v1 , . . . , vR . We may take more than one strategic goal into consideration; doing so is expressed by using more than one weight vector, an approach that produces different strategic objective functions. Constraints represent the following conditions:

A MCDSS for competence-driven project portfolio selection

5

• The overall real work time of each employee j in each period t is limited by his/her capacity ajt . Formally: K X R X

xkjrt ≤ ajt

∀j, t

(2.4)

k=1 r=1

• The sum of ideal work times (i. e., real work times multiplied by corresponding efficiencies) invested into each work package (k, r) amounts to the required effort dkr for this work package. Formally: m T X X t=1 j=1

γjrt xkjrt = dkr

n X

cik yi

∀k, r,

(2.5)

i=1

where cik is 1 if task k belongs to project i and 0 otherwise • Work can be invested into a task k only within the time interval between its ready time ρk and its due date δk . Formally: (t − ρk )xkjrt ≥ 0 and (δk − t)xkjrt ≥ 0

∀k, j, r, t

(2.6)

2.2. Solution techniques In Ref. 10, a scalarized version of the problem defined in the previous subsection, obtained by taking a weighted mean of the objective functions (2.2) and (2.3) as the single overall objective, has been solved by a combination of a metaheuristic solution technique on the upper decision level and a greedy heuristic on the lower decision level. Both a Genetic Algorithm (GA)14 and an Ant Colony Optimization (ACO) algorithm15 were investigated for the metaheuristic approach. Experimental results showed GA’s superiority over ACO for those cases in which no additional constraints were present, and ACO’s superiority over GA in those cases where additional constraints, e. g. restricting the number of employees per task or defining minimum and/or maximum numbers of projects from certain subsets, were imposed. In Ref. 11, the investigation was extended to the multiobjective case where (2.2) and (2.3) were treated separately and Pareto-optimal sets were determined. Instead of GA and ACO, corresponding multiobjective metaheuristic solution procedures NSGA-II16 and P-ACO17 , respectively, were applied. For the special case involving only two objectives, the lower-level decision problem was solved exactly (using CPLEX) instead of heuristically in order to avoid quality losses which occur when applying the greedy heuristic procedure. The comparison between the GA-based and the ACO-based approach yielded similar trends as in the single-objective case. Some real-life problems may have too many efficient solutions even for metaheuristic procedures to generate within a reasonable runtime. This situation may be tackled by one of the following measures or combinations thereof. First, a screening procedure may be employed to reduce the number of project proposals by identifying those that are worthy of further consideration. This can be achieved by means of a simple scoring tool (e.g., the one proposed in Ref. 18) which is quantitative

6

C. Stummer, E. Kiesling & W. J. Gutjahr

enough to possess a certain degree of rigor, yet is not so complex as to discourage users, and does not require detailed preference data.19 Second, planning periods may be extended (e.g., doubled to a length of two months instead of one month) which considerably reduces the scheduling effort in the innermost loop of the optimization problem. Third, the employees may be partitioned into research groups to which appropriate project proposals are assigned (for an related approach see 20). After solving the corresponding (smaller) problems for each separate research group, identified (sub-) portfolios can be assembled to overall portfolio solutions. Fourth, the generation of nondominated solutions may be embedded into the interactive solution procedure. Starting with a number of efficient portfolios that have been computed with respect to some quasi-random criteria weights21 the DM’s actions (e.g., setting aspiration levels) are then used to guide the search for additional nondominated solutions only in promising areas rather than in the whole efficient set. All the above modifications can be easily implemented. However, they come at the price of missing efficient portfolios because (i) some projects have been falsely deemed to be unattractive during screening, (ii) projects are scheduled later than necessary because preceding projects are not recognized as completed before the end of a too lengthy planning period and/or (iii) synergy effects from collaboration of researchers from several groups may be lost if project proposals are exclusively assigned to such a group. Further, the drawback of computing only some efficient portfolios at the beginning and more during the interactive search lies in reduced usability as, for instance, it may take at least a few seconds after each activity before additional portfolios have been found.

3. The Interactive Decision Support Tool Numerous approaches to multicriteria decision support have been proposed since the 1970s. Early mainframe systems focused mainly on mathematical programming and were often of a prototypical nature, lacking a user-friendly interface.22 During the 1980s, research interest in the behavioral aspects of multicriteria decision making grew, fueling work on interactive methods. Starting from the mid 1980s, advanced graphical methods became technically viable and Visual Interactive Modeling, which had started in the late 1970s,23 began to appear more frequently on the research agenda, resulting in the creation of several MCDSS emphasizing graphical presentation in the late 1980s and early 1990s (cf., e. g., Refs. 24, 25, 26, 27, 28, 29, 30, 31). Since then, research interest in interactive graphical methods in MCDSS appears to have leveled off, while the application of MCDSS in various fields became an increasingly active topic (e. g., Refs. 32, 33, 34, 35, 36, 37). The interactive decision support tool for competence-driven project portfolio selection presented in this paper is in line with this stream of research. In addition to the innovative application, it also introduces new interactive approaches and applies established graphical methods in novel ways. Our tool does not require any explicit knowledge of the properties of the DM’s utility function, but rather employs

A MCDSS for competence-driven project portfolio selection

7

a two-phase procedure that first identifies efficient portfolios and then allows the DM to explore the solution space and find the individually preferred portfolio. The preceding Sections 2.1 and 2.2 have provided the formulation of, as well as a description of the proper solution techniques for, the underlying multicriteria mathematical optimization problem. The remainder of this section focuses on the subsequent interactive phase of our MCDSS. If the number of non-dominated portfolios is small, the DM might be capable of comparing all candidate portfolios simultaneously. However, for problem instances of a realistic size, the number of efficient solutions may frequently become rather large (e. g., 36 as in our sample application described in Section 4 or even several hundred portfolios), clearly exceeding the DM’s capacity for simultaneous comparison. Our MCDSS therefore offers various mechanisms for exploring available alternatives in a step-by-step procedure. The most basic representation of the criteria space offered by the MCDSS is a list of efficient portfolios and their respective criteria values in tabular form. The DM can sort the list by any criterion and select individual portfolios to obtain more detailed information, such as executed projects, scheduling, staff assignments, workload statistics, etc. While tabular reports are not generally inferior to graphical representations regarding readability, interpretation accuracy and decision quality,38 we found that tabular data alone is not particularly helpful for the decision-maker when solving the problem at hand. Here, graphical visualizations offer valuable support as they facilitate the simultaneous presentation of vast amounts of information along several dimensions of the problem. We therefore investigated the applicability of various alternative techniques for visualizing multidimensional data that have been suggested in the literature so far. A first group of graphical methods we were looking at aims at representing n-dimensional points as dissimilar objects – such as faces39 , houses27 , trees or castles40 , as well as glyphs41 – whose individual attributes correspond with the respective criteria values. However, it turned out to be difficult if not impossible at all to design user-friendly mechanisms that allow for the interactive manipulation of aspiration levels within these visualizations. We also considered the use of Andrews plots42 , that represent each alternative as a line defined by a trigonometric function parametrized with the values achieved in the various dimensions. The method is useful for detecting groups of similar points (i. e., clustering), but requires the dimensions to be ordered by importance, because lower frequency terms are easier to interpret from the graph than higher frequency ones. Since we have opted for keeping the amount of required a priori preference information as low as possible it was decided not to follow up with this representation as well. Interactive Decision maps43, 44 are another interesting visualization technique that can be applied in the framework of multi-criteria optimization. In such maps, several efficiency frontiers of two criteria are depicted depending on the value of the third criterion. The interpretation of the resulting curves largely corresponds to the interpretation of height curves in a topographical map, which facilitates assessment. If the number

8

C. Stummer, E. Kiesling & W. J. Gutjahr

of criteria that needs to be considered simultaneously is greater than three, several maps can be depicted in parallel. In the case of four criteria, these maps can be aligned by ascending value of the fourth criterion. When five criteria need to be considered, a matrix of decision maps may be displayed. Although in principle, five or an even greater number of criteria can be considered simultaneously using this method (e.g. by animating the maps), this comes at the price of an increased cognitive load imposed on the decision maker. If the number of criteria is large, it will be necessary to fix some of them and limit the analysis to a reduced subset. Another disadvantage of this method is that it does not represent all criteria in a homogeneous way. If five criteria are considered, two are given on axes, two are represented as cells in a matrix, and one criterion is given by color shading. Choosing appropriate criteria to be represented by color shading or as cells would be far from trivial in the context of competence-driven project portfolio selection, as the levels obtained will typically differ in each criterion of each efficient portfolio. While this problem could be overcome by categorizing the levels, this would require the DM to make an important distinction between criteria when deciding which of them to show on axes and as shades/cells, respectively, which we deem undesirable. After an in-depth analysis of remaining approaches, as well as several discussions with representatives of the EC3, we finally implemented interactive visualizations for portfolio selection based on heatmaps, parallel coordinate plots, column charts and a so-called “competence map”. All alternative views are linked together, inasmuch as the DM can seamlessly switch between them while manipulations made in one perspective become visible immediately in other perspectives as well. A default sequence of these views is suggested by the MCDSS, starting with the heatmap-interface, which is helpful in cases involving numerous portfolio alternatives, followed by parallel coordinate plots and column charts. The competence map-interface finally makes it possible to trace the development of the employees’ competencies for a single portfolio over time. Note that our observations when applying the MCDSS have supported our initial assumption that there is no single best graphical method for representing the solution space and interacting with it.

3.1. Heatmap Heatmaps are typically used in molecular biology and clinical applications (e. g. for DNA sequence visualization),45, 46 but are less well-known and less frequently applied in other fields. The term arises from the red to blue color scale that is often used and appears to be the most commonly mentioned in the relevant literature. Cook et al.45 suggest the term “colored matrix plot” as a more appropriate name, as this kind of visualization is basically a matrix where the cells are colored according to the cell value. In a project portfolio selection context, we suggest using the rows in the matrix to represent portfolios and columns for each criterion. An example is provided in Fig. 1. The individual cells are colored according to the relative numerical value

A MCDSS for competence-driven project portfolio selection

9

achieved by the portfolio in the respective criterion. The relative scale ranges from the poorest (lowest color intensity) to the best performing (highest color intensity) portfolio in each criterion. Clicking the column title labels, the portfolios can easily be sorted by ascending or descending criterion values, which gives the DM a first impression about the distribution of values and may reveal interesting patterns. The user can choose from several color mappings, ranging from simple monochromatic to complex trichromatic (e. g., the red-yellow-green “stop light” scheme). To exclude portfolios, the DM may impose constraints on each criterion by specifying reservation levels as upper or lower limits, thus excluding all portfolios that do not achieve a value in the desired interval. This can easily be achieved by right-clicking a cell and selecting “Set as minimum” or “Set as maximum” from the context menu, respectively.

Fig. 1. Heatmap

10

C. Stummer, E. Kiesling & W. J. Gutjahr

3.2. Parallel coordinate plot Heatmaps provide a proper first overview but they are not the best way to visualize the distribution of values due to their lack of geometric interpretability.45 A parallel coordinate plot is more suitable both to grasp the distribution of values and to make more detailed comparisons from a smaller set of individual portfolios.47, 48 In parallel coordinate plots, criteria are represented on separate axes that are laid out in parallel. A portfolio is then depicted as a profile line that connects the points marking values achieved in each criterion. When multiple portfolios are selected, the profile lines are superimposed, which allows for convenient comparison. A relative scale where zero represents the value achieved by the portfolio performing poorest in the respective criterion is used on each axis. The markings for each portfolio then indicate the percentage “gain” with respect to this “weakest” portfolio (i. e., a value of 100 means that the score of the portfolio is twice as high as that of the portfolio with the lowest value in this criterion). Fig. 2 provides an example. Patterns such as positive or negative correlations can easily be identified in criteria laid out next to each other. To enable the DM to interactively explore trade-offs and state his/her preferences, an interval bar is rendered on each axis. Using the mouse-pointer to click the top or bottom region of this bar, the user can move the upper and/or lower bounds (i. e., reservation levels) by dragging the mouse. While a dragging operation is under way, the profile lines of all portfolios that remain in the candidate set at the current bar position are marked green while all excluded portfolios are marked red, which immediately gives the DM feedback about implicit sacrifices in other criteria. 3.3. Interactive column chart The most accurate, but also least holistic, perspective on the set of efficient portfolios is provided by an extended column chart, which shows the absolute values achieved by one or several selected portfolios as columns on multiple parallel axes; Fig. 3 demonstrates such a chart for a single portfolio with index 3, represented by green columns indicating the level achieved for each criterion (exact numbers are displayed as soon as the mouse cursor points at a bar). By selecting multiple portfolios, the DM benefits from receiving more detailed information on the pattern of the criteria values in the selected portfolios, allowing him/her to compare them directly against each other. However, this comes at a price: the number of portfolios that should be put side-by-side in one chart is typically limited to about five or seven, as it becomes quite difficult to properly distinguish between the columns if more than seven are displayed. Other than that, the mechanism of further restricting or expanding the set of remaining portfolios works similarly to that described above for the parallel coordinate plot. Once again, (blue) interval bars are used to control constraints imposed on each criterion. While dragging the upper or lower limit of an interval bar, red areas in other interval bars provide the DM with immediate visual feedback about implicit tradeoffs to be made.

A MCDSS for competence-driven project portfolio selection

11

Fig. 2. Interactive parallel coordinate plot

These areas show changes in the range of achievable values due to changes in the set of candidate portfolios fulfilling the requirements imposed by the DM. Thus, while moving some minimum or maximum limits in one criterion, the effect of the DM’s current choice on all other criteria are shown. When the DM completes a dragging operation by releasing the mouse button, the target intervals of all criteria are adapted so that they range from the lowest to the highest value of the remaining solutions. Additionally, green horizontal marks for those portfolios that are still in the candidate set and gray marks for excluded portfolios, each representing a value achieved by a portfolio, help the DM to grasp the distribution of values.

3.4. Competence map All techniques described so far support the DM in exploring the solution space and identifying preferred candidate portfolios based on their objective values. In addition, techniques are required that provide deeper insight through visualizing the prospective competence development within an organization for single portfolios. To this end, we have developed a novel method, i.e. the so-called competence map. It extends the heatmap concept to the employees’ level and, furthermore, allows the visualization of competence development over time. Here, rows are referring to

12

C. Stummer, E. Kiesling & W. J. Gutjahr

Fig. 3. Interactive column chart

employees while columns still are referring to competencies. Cells are colored with respect to the competence value of the respective employee, with a higher color intensity corresponding to a higher competence level. Fig 4 shows a snapshot for an example with ten employees and five groups of competencies labeled I to V. In using the competence map-view the DM not only is provided with a good overview of current competence levels and their distribution within the organization, but also may track the expected development for the currently selected portfolio in a quite holistic, yet also easily comprehensible way. To this end, the DM can choose to play an animation showing the evolution of competencies to the planning horizon, analyze the development “frame by frame” (i. e., period by period) or select individual periods from a drop-down box. Thus, the distribution of competencies among employees becomes apparent, revealing whether certain competencies are highly concentrated (i. e., there are few highly competent experts) or distributed more evenly. This visualization may provide an overview of the current state as well as of the possible future development of human capital within the organization, yielding valuable insights for the strategic management of competencies. In some situations, the DM might be more interested in competence gains and losses over time than in absolute values. To gain a clearer understanding of these

A MCDSS for competence-driven project portfolio selection

13

Fig. 4. Competence map [animated]

relative changes and make better judgments about their strategic implications, the DM then may switch the map to a relative scale. In this mode, green cells signal competence gains and red cells show areas in which competence is lost while the color intensity corresponds to the extent of gains/losses with respect to the initial competence level. 4. Sample Application This section describes the application of our MCDSS in a real-world setting provided by the Electronic Commerce Competence Center (EC3) Austria. The EC3 is a public-private partnership institution funded by the Austrian Federal Ministry of Economic Affairs and the City of Vienna, as well as by twelve private enterprises (e. g., T-Mobile, Tiscover, Swarovski Crystal Online). By embedding innovation practices into a collaborative network consisting of both the three major universities in Vienna (i. e., the University of Vienna, the Vienna University of Technology and the Vienna University of Economics and Business Administration) and the company partners, the EC3 strives to implement a fast and problem-tailored transfer of knowledge into its business partners’ realm of production and value generation. To this end, some 15 FTEs (full time equivalents) of permanent research staff are assigned to four working groups dealing with (i) the creation, exploration and understanding of large information spaces, (ii) logical models, designs, and mechanisms of interoperable Web-based systems, (iii) empirical business analyses using formal quantitative methodologies and modeling techniques, and (iv) the evaluation of business ideas and models, including empirical analysis of customer needs and further methods of market research.

14

C. Stummer, E. Kiesling & W. J. Gutjahr

As a first step in problem formulation, competencies relevant at the EC3 had to be identified. A fine-grained taxonomy encompassing 80 professional and methodological competencies organized into nine classes was therefore elaborated through a multi-level definition process involving all of the organization’s employees.49 Simultaneously, a model for measuring current competence levels was developed. This model combines 56 competence indicators based on both objective evidence in terms of formal qualifications and professional experience and subjective evidence, viz. competence ratings by peers, the scientific director, and the researcher him-/herself. A total of 28 researchers – including the heads of the research groups, the scientific director, and several freelancers, in addition to the 15 permanent researchers – were surveyed via e-mail in order to collect the objective and subjective evidences. The competence score for each researcher in each competence was then computed by summing up the contributions of all objective evidences a researcher holds, adding a score derived from the subjective competence ratings and normalizing the result to the interval [0,100]. Learning and depreciation rates were defined assuming that learning by experience is faster than depreciation (for a discussion cf. Refs. 50, 51, 52). For the time being, no distinctions were made between the competencies, except for several methodological competencies that were supposed to grow and diminish more slowly. Data on 18 potential projects (two of which consisted of two tasks, while the others involved only one task each) was gathered from project plans and through estimating the effort required in each competence. Nine different competencies were required per task and the project duration was about 12 months on average. The planning horizon was set at 24 months. The amount of third-party funding was provided as a measure of economic gain. Researchers’ disposable capacities were estimated and additional constraints limiting the maximum number of researchers per task were introduced. For the sample application at the EC3 six objectives were selected: two monetary (viz. generated partner value, third-party funding received) and four competence objectives, each of which is a weighted combination of ten (out of a total number of 80) individual competencies (cf. sets of Eqs. (2.2) and (2.3), respectively). The problem definition phase resulted in an input file that can be processed by the MCDSS. Using a Pentium M 760 (2GHz), a Pareto front was established in less than 5 minutes using the meta-heuristic solution procedures described in Section 2.2, or after 15 minutes when performing a complete enumeration of all portfolios. In both cases, an identical set of 36 efficient Pareto-efficient portfolios was identified. This is a comparatively small number, given that randomly generated test instances of similar size with respect to the number of projects and competencies yielded several hundred efficient portfolios. Here, the small number may be attributed to the relative homogeneity in competence requirements of candidate projects, to correlation in objectives, and to the very low variation in two of the objectives (viz. software engineering and eBusiness Models). This low variation is caused by a lack of available candidate projects requiring the respective competencies; this finding

A MCDSS for competence-driven project portfolio selection

15

by itself may be of interest to the DM. Upon termination of the solution procedure, the result is automatically opened in the MCDSS tool for analysis.

Fig. 5. Sample application: initial screen

Fig. 5 shows the initial screen presented to the DM. On the left-hand side, a table lists all of the efficient project portfolios found and the respective objective scores achieved. On the right-hand side, interactive visualizations described in Section 3 are presented to the DM. By default, the heatmap perspective is displayed when opening a file, as it is commonly the most suitable type of visualization to start with. As noted earlier, the list of portfolios is linked with all graphical representations, which allows the DM to select a portfolio from the list to highlight or select it in the interactive diagrams. Vice versa, when interacting with the graphics, the table is continuously updated to reflect the impact of the DM’s actions. A minimum amount of third-party funding is considered critical by the DM. He therefore sorted the portfolios by ascending values by clicking the criterion label in the chart and imposed a minimum requirement by right-clicking a cell representing a value that is just acceptable and selecting “Set as minimum”. The list of candidate portfolios was thereby shortened from 36 to 24. A closer inspection of the resulting heatmap depicted in Fig. 6 makes clear that the same competence level is reached in the Software Engineering objective in most remaining portfolios. The reason for this might lie in the fact that there are few projects that require this competence, or that the existing competence levels are too low to execute most projects that require it. In either case, the opportunity cost of maximizing “Software Engineering” competence is high as the development

16

C. Stummer, E. Kiesling & W. J. Gutjahr

Fig. 6. Sample application: reduced heatmap

of other competence objectives is rather poor in the portfolio that maximizes it. One can also observe that the “eBusiness Models” score achieved is the same in all remaining portfolios except one, which requires the DM to trade off third-party funding and this competence. After imposing minimum requirements considered indispensable by the DM, he/she can use the parallel coordinate chart to gain a better understanding of the distribution of values in the remaining portfolios, thus getting a better picture of the range of available alternatives. Switching the perspective by clicking a toolbarbutton and selecting all portfolios resulted in the screen shown in Fig. 7. The large number of remaining portfolios causes overplotting, which makes it hard to identify the lines of individual portfolios even though the line below the mouse pointer is highlighted. However, at this stage of the decision-making process,

A MCDSS for competence-driven project portfolio selection

17

Fig. 7. Sample application: interactive parallel coordinates plot

a holistic view of the solution space is more valuable than detailed comparisons of individual options. The DM can drag the blue interval bars to interactively explore interdependencies and tradeoffs between criteria. The effects of the action currently considered (indicated by the mouse pointer position during a dragging operation) are displayed in real-time by marking all lines of portfolios that would currently be excluded from the candidate set red while leaving all others green. Our DM considered imposing a constraint on the “Data Science” competence level and, thus, received immediate feedback on how this action will impact other criteria (see Fig. 8). After the DM released the mouse button, 16 candidate portfolios remained. Having already reduced the number of candidate portfolios substantially, the DM switched to the more detailed, but less holistic interactive column chart, which shows the values achieved by individual, selected portfolios as columns. While similar to the parallel coordinate plot, this type of visualization makes it easier for the DM to compare individual portfolios, but still allows him/her to experiment with constraints. Fig. 9 shows the screen while the DM was dragging the interval bar for criterion “Partner Value”. Releasing the mouse button at the current position reduced the set of candidate portfolios to only five. Note that the y-axes do not have a common scale, as this chart depicts the absolute values, which are not comparable across criteria. To avoid cluttering up the diagram, individual axes for each criterion are not shown in Fig. 9. With only few portfolios remaining, each of them can be inspected more closely. In particular, the DM may compare assignments of researchers to projects, their

18

C. Stummer, E. Kiesling & W. J. Gutjahr

Fig. 8. Sample application: interactive parallel coordinates plot – dragging operation

individual workloads, the distribution of competence gains among them and temporal differences in the competence development. Assignments and workload are displayed as when the DM double-clicks a portfolio. Fig. 10 shows the state of competence development in one of the remaining portfolios in the final phase. (Note that the names of researchers have been made anonymous.) During this phase, the DM has opted to show changes in competence instead of absolute values in order to get an impression of the organization’s relative competence development. This visualization also illustrates whether gains are distributed evenly among researchers and competencies, or if they are concentrated in particular areas. 5. Conclusions While project portfolio selection already constitutes a challenging decision problem, our paper highlights two additional factors that contribute to the complexity of the problem at hand: (i) scientific staff has to be properly assigned, which has a particularly strong influence on the scheduling in the short-run and the competence development in the long-run, and (ii) schedules for the selected projects have to be generated considering both financial and competence-related resource constraints. We tackle this problem through a two-step procedure. Firstly, a multicriteria mathematical programming approach has been formulated that allows to determine efficient project portfolios (if necessary, by means of meta-heuristic solution procedures). In the main part of this paper, we then have focused on providing visualization and interactivity in the succeeding selection phase. To this end,

A MCDSS for competence-driven project portfolio selection

19

Fig. 9. Sample application: interactive column chart – dragging operation

our MCDSS offers several alternative interfaces capable of supporting managers in finally finding their individually “best” portfolios. Feedback from the system’s practical application at the EC3 suggests that the MCDSS provides valuable insights and offers a rational basis for the strategic selection of research projects. Regarding limitations, the hardest one we experienced was related to the availability and quality of input data. However, this difficulty is not unique for our approach, but rather holds for all other approaches as well; nevertheless, it confines the applicability of formal decision support approaches to those cases in which the necessary information is available. It should also be mentioned that the explicit assessment of competencies within an organization is a challenging task that may easily face resistance or opposition from the side of the employees and can, if done in an incautious way, turn out as a source of considerable tension. We took this point very serious. In our opinion, an MCDSS of the proposed type can only be expected to be proli?c if the employees recognize advantages for themselves in its application so that they are interested in its well-functioning. We investigated this question in detail in a (partially empirical) study Ref. 53. Main results were that the opportunity of participation in the project proposal process can be a major incentive to support a system of the mentioned kind, and that the data privacy

20

C. Stummer, E. Kiesling & W. J. Gutjahr

Fig. 10. Sample application: competence map [animated]

issue has to be given considerable importance when implementing the system. Deriving “better” data in terms of both quantity and quality (e.g., by means of decision conferencing54 ) therefore constitutes a valuable field for further research. With respect to low quality data and uncertainty of project data ideas from robust portfolio modeling55, 56 or contingent portfolio programming57 , respectively, may be considered; for a review on more recent developments in research on resource allocation decisions see Ref. 58. Next, ideas from group decision-making59 and negotiation analysis60 might be integrated in order to elicit preferences from multiple stakeholders and to propose proper (e. g., fair) compromises. As far as the graphical interfaces are concerned, their respective advantages and limitations (e. g. regarding interpretation accuracy, decision speed, the DM’s confidence in his/her decision etc.) and their applicability in various phases of the decision process needs experimental investigation. Finally, aspects such as interactive problem formulation, data management and sensitivity analysis in the MCDSS – all of which we have largely disregarded – still need to be addressed. Acknowledgments The authors would like to thank Rudolf Vetschera for helpful comments on an earlier version of this paper as well as Karl Fr¨oschl, Research Director at the EC3, for valuable discussions throughout the implementation of our MCDSS. References 1. D. A. Nembhard, Heuristic approach for assigning workers to tasks based on individual learning rates, International Journal of Production Research 39(9) (2001) 1955–1968.

A MCDSS for competence-driven project portfolio selection

21

2. M. Yoshimura, Y. Fujimi, K. Izui and S. Nishiwaki, Decision-making support system for human resource allocation in product development projects, International Journal of Production Research 44(5) (2006) 831–848. 3. M. A. Coffin and B. W. Taylor, Multiple criteria R&D project selection and scheduling using fuzzy logic, Computers & Operations Research 23(3) (1996) 207–220. 4. R. Kolisch, K. Meyer and R. Mohr, Maximizing R&D portfolio value, ResearchTechnology Management 48(3) (2005) 33–39. 5. H. Sun and T. Ma, A packing-multiple-boxes model for R&D project selection and scheduling, Technovation 25(11) (2005) 1355–1361. 6. R. Venkatraman and S. Venkatraman, R&D project selection and scheduling for organizations facing product obsolescence, R&D Management 25(1) (1995) 57–70. 7. K. Heidenberger and C. Stummer, Research and development project selection and resource allocation: a review of quantitative modelling approaches, International Journal of Management Reviews 1(2) (1999) 197–224. 8. A. Tsoukias, From decision theory to decision aiding methodology, European Journal of Operational Research 187(1) (2008) 138–161. 9. J. Hodgkin, V. Belton and A. Koulouri, Supporting the intelligent MCDA user: a case study in multi-person multi-criteria decision support, European Journal of Operational Research 160(1) (2005) 172–189. 10. W. J. Gutjahr, S. Katzensteiner, P. Reiter, C. Stummer and M. Denk, Competencedriven project portfolio selection, scheduling and staff assignment, Central European Journal of Operations Research 16(3) (2008) 281–306. 11. W. J. Gutjahr, S. Katzensteiner, P. Reiter, C. Stummer and M. Denk, Multi-objective decision analysis for competence-oriented project portfolio selection, Tech. Rep. 200802, ISDS, University of Vienna (2008). 12. A. N. Chen and T. M. Edgington, Assessing value in organizational knowledge creation: considerations for knowledge workers, MIS Quarterly 29(2) (2005) 279–309. 13. O. Ngwenyama, A. Guergachi and T. McLaren, Using the learning curve to maximize IT productivity: a decision analysis model for timing software upgrades, International Journal of Production Economics 105(2) (2007) 524–535. 14. Z. Michalewicz, Genetic Algorithms + Data Structures = Evolution Programs (Springer, London, 1996). 15. M. Dorigo and T. St¨ utzle, Ant Colony Optimization (MIT Press, Cambridge, 2004). 16. K. Deb, A. Pratap, S. Agarwal and T. Meyarivan, A fast and elitist multiobjective genetic algorithm: NSGA-II, IEEE Transactions on Evolutionary Computation 6(2) (2002) 182–197. 17. K. F. Doerner, W. J. Gutjahr, R. F. Hartl, C. Strauss and C. Stummer, Pareto ant colony optimization: a metaheuristic approach to multiobjective portfolio selection, Annals of Operations Research 131(1) (2004) 79–99. 18. A. D. Henriksen and S. W. Palocsay, An Excel-based decision support system for scoring and ranking proposed R&D projects, International Journal of Information Technology & Decision Making 7(3) (2008) 529–546. 19. A. D. Henriksen and A. J. Traynor, A practical R&D project selection scoring tool, IEEE Transactions on Engineering Management 46(2) (1999) 158–170. 20. K. F. Doerner, W. J. Gutjahr and P. C. Nolz, Multi-criteria location planning for public facilities in tsunami-prone coastal areas, OR Spectrum, forthcoming. 21. P. Bratley and B. L. Box, Algorithm 659: implementing Sobol’s quasirandom sequence generator, ACM Transactions on Mathematical Software 14(1) (1988) 88–100. 22. P. Korhonen, Interactive methods, in Multiple Criteria Decision Analysis: State of the Art Surveys, eds. J. Figueira, S. Greco and M. Ehrgott (Springer, Berlin, 2004) pp.

22

C. Stummer, E. Kiesling & W. J. Gutjahr

641–666. 23. R. D. Hurrion, An investigation of visual interactive simulation methods using the jobshop scheduling problem, Journal of the Operational Research Society 29(11) (1978) 1985–1093. 24. V. Belton and S. Vickers, V.I.S.A. - VIM for MCDA, in Improving Decision Making in Organizations, eds. A. Lockett and G. Islei (Springer, Berlin, 1989) pp. 287–304. 25. J. N. Cl´ımaco and C. Henggeler Antunes, Implementation of a user-friendly software package: a guided tour of TRIMAP, Mathematical and Computer Modelling 12(10/11) (1989) 1299–1309. 26. R. Klimberg, GRADS: a new graphical display system for visualizing multiple criteria solutions, Computers and Operations Research 19(7) (1992) 707–711. 27. P. Korhonen, Using harmonious houses for visual pairwise comparison of multiple criteria alternatives, Decision Support Systems 7(1) (1991) 47–54. 28. P. J. Korhonen and J. Laakso, A visual interactive method for solving the multiple criteria problem, European Journal of Operational Research 24(2) (1986) 277–287. 29. A. Lewandowski and J. Granat, Dynamic BIPLOT as the interaction interface for aspiration based decision support systems, in Multiple Criteria Decision Support, eds. P. Korhonen, A. Lewandowski and J. Wallenius (Springer, Berlin, 1991) pp. 229–241. 30. W. Michalowski and T. Szapiro, A bi-reference procedure for interactive multiple criteria programming, Operations Research 40(2) (1992) 247–258. 31. C. Strauß and C. Stummer, Multiobjective decision support in it-risk management, International Journal of Information Technology & Decision Making 1(2) (2002) 251– 268. 32. P. J. Agrell, A. Stam and B. J. Lence, Second-generation interactive multi-criteria decision making in reservoir management, in Trends in Multicriteria Decision Making, eds. T. J. Stewart and R. C. Van Den Honert (Springer, Berlin, 1997), pp. 324–336. 33. D. Borenstein, A visual interactive multicriteria decision analysis model for FMS design, International Journal of Advanced Manufacturing Technology 14(11) (1998) 848–857. 34. T. Hanne and H. L. Trinkhaus, knowcube for MCDM: Visual and interactive support for multicriteria decision making, Tech. Rep. 50, Fraunhofer ITWM (2003). 35. J. Mustajoki, R. H¨ am¨ al¨ ainen and K. Sinkko, Interactive computer support in decision conferencing: two cases on off-site nuclear emergency management, Decision Support Systems 42(4) (2007) 2247–2260. 36. C. Thieke, K.-H. K¨ ufer, M. Monz, A. Scherrer, F. Alonso, U. Oelfke, P. Huber, J. Debus and T. Bortfeld, A new concept for interactive radiotherapy planning with multicriteria optimization: first clinical evaluation, Radiotherapy & Oncology 85(2) (2007) 292–298. 37. C. Theetranont, P. Haddawy and D. Krairit, Integrating visualization and multiattribute utility theory for online product selection, International Journal of Information Technology & Decision Making 6(4) (2007) 723–750. 38. G. W. Dickson, G. DeSanctis and D. J. McBride, Understanding the effectiveness of computer graphics for decision support: a cumulative experimental approach, Communications of the ACM 29(1) (1986) 40–47. 39. H. Chernoff, The use of faces to represent points in k-dimensional space graphically, Journal of the American Statistical Association 68(342) (1973) 361–368. 40. B. Kleiner and J. A. Hartigan, Representing points in many dimensions by trees and castles, Journal of the American Statistical Association 76(374) (1981) 260–269. 41. E. Anderson, A semigraphical method for the analysis of complex problems, Technometrics 2(3) (1960) 387–391.

A MCDSS for competence-driven project portfolio selection

23

42. D. Andrews, Plots of high-dimensional data, Biometrics 28(1) (1972) 125–136. 43. A. Lotov, V. Bushenkov and A. Chernov, Internet, gis and interactive decision maps, Journal of Geographic Information and Decision Analysis 1(2) (1997) 118–149. 44. A. V. Lotov, V. A. Bushenkov and G. K. Kamenev, Interactive Decision Maps, Approximation and Visualization of Pareto Frontoier (Kluwer, Dordrecht, NL, 2004). 45. D. Cook, H. Hofmann, E.-K. Lee, H. Yang, B. Nokolau and E. Wurtele, Exploring gene expression data, using plots, Journal of Data Science 5(2) (2007) 151–182. 46. M. B. Eisen, P. T. Spellman, P. O. Brown and D. Botstein, Cluster analysis and display of genome-wide expression patterns, Proceedings of the National Academy of Sciences of the United States of America 95 (1998) 14863–14868. 47. A. Inselberg, The plane with parallel coordinates, The Visual Computer 1(4) (1985) 69–91. 48. E. J. Wegman, Hyperdimensional data analysis using parallel coordinates, Journal of the American Statistical Association 85(411) (1990) 664–675. 49. M. Denk and K. Fr¨ oschl, Datenstrukturen und Erhebungsmethodik zur Bewertung von Kompetenzen, Internal Report, EC3 [German], (2007). 50. A. A. Alamri and Z. T. Balkhi, The effects of learning and forgetting on the optimal production lot size for deteriorating items with time varying demand and deterioration rates, International Journal of Production Economics 107(1) (2007) 281–294. 51. M. Y. Jaber and H. V. Kher, Variant versus invariant time to total forgetting: the learn-forget curve model revisited, Computers & Industrial Engineering 46(4) (2004) 697–705. 52. M. Y. Jaber and S. Sikstr¨ om, A numerical comparison of three potential learning and forgetting models, International Journal of Production Economics 92(3) (2004) 281–294. 53. K. Froeschl, M. Denk, W. Gutjahr, A. Jerusalem, H. Riedman and C. Stummer, Project selection, staffing and competence building: an integrated approach to human resource management, tech. rep., ISDS, University of Vienna (2008). 54. L. D. Phillips and C. A. Bana e Costa, Transparent prioritisation, budgeting and resource allocation with multi-criteria decision analysis and decision conferencing, Annals of Operations Research 154(1) (2007) 51–68. 55. J. Liesi¨ o, P. Mild and A. Salo, Preference programming for robust portfolio modeling and project selection, European Journal of Operational Research 181(3) (2007) 1488– 1505. 56. J. Liesi¨ o, P. Mild and A. Salo, Robust portfolio modeling with incomplete cost information and project interdependencies, European Journal of Operational Research 190(3) (2008) 679–695. 57. J. Gustafsson and A. Salo, Contingent portfolio programming for the management of risky projects, Operations Research 53(6) (2005) 946–956. 58. D. N. Kleinmuntz, Resource allocation decisions, in Advances in Decision Analysis, eds. E. Ward, R. E. Miles and D. von Winterfeldt (Cambridge University Press, Cambridge, 2007) pp. 400–418. 59. A. Salo and J. Liesi¨ o, A case study in participatory priority setting for a scandinavian research program, International Journal of Information Technology & Decision Making 5(1) (2006) 65–88. 60. M. Tavana and D. T. Kennedy, N-site: A distributed consensus building and negotiation support system, International Journal of Information Technology & Decision Making 5(1) (2006) 123–154.

View publication stats