dbnet.home Bookmark and Share

MEET FEATURE REQUIREMENTS, SCHEDULE, and BUDGET! Consulting available to teach your organization to apply this methodology and tools for powerful quantitative management of your business processes.

  • Instruction of your staff at your site, using courseware, application example, and a functional template.
  • Mentoring your key staff on an ongoing or periodic basis, on these tools or on Quantitative Program Management.
  • Contracting or Employment in your organization on specific topics.

Brochure  ...  Contact for Details

Check out my YouTube Channel: Power Operational Intelligence

Now Live! Overview, Structure, Task Data, Table Design, SQL Server, and Re-Linking now showing.

Video courses covering material on this website, and more, are presented in the playlists.

Code snippet links at YouTube Code Snippets. Twitter at @poweroperation1, #poweropi, #poweroperationalintelligence.

Subscribe on YouTube, and click the "Notification" Bell icon to be notified as content is published.


Using Excel with a Database

Flat Excel spreadsheets, some expressed as tables, are commonly used to represent, analyze and manage business problems. But there are more powerful ways to use Excel: connecting Excel Tables to a Database environment, to add the power of QUERYs and JOINs to combine the Tables to drive Pivot reports, Pivot charts, QUERY Tables, and PowerBI graphical elements.

This capability has been available for years using MS Access and SQL Server. In recent years, Microsoft have added PowerQuery to Excel, as well as PowerBI Desktop, with intent to make Excel-with-a-database for Business Intelligence ("BI") a self-service proposition, available to ordinary mortals. This website is about Power Operational Intelligence ("PowerOpI"), a use of Excel as a data source (among others) initially developed with MS Access and SQL Server, and extended to PowerQuery.

These are explored as tools for Program Management of complex projects, in posts on Silicon Valley Project Management, and collected here as well as on this site for convenient viewing on this site.

PowerOpI provides automation for local and multi-user distributed use, supporting:

  • Creating and editing data connections between Excel and database environments, including creation and editing of Connection Strings to databases and use of configured QUERYs as follows. PowerOpI supports:
  • Environments for creating and editing QUERYs in Excel for SQL (used with MS Access and SQL Server), and for M (used with PowerQuery). These QUERYs can be conveniently configured to incorporate related file structure and related QUERYs to support distributed use and to minimize maintenance required in distributed cases;
  • Examples and templates for Connections, Connection Strings, QUERYs, Source Data Structured Tables in Excel, and Pivot Table Reports in Excel. Examples of Source Data and resulting Pivot Tables are focused on Program Management, but can be extended to other uses as well.

PowerOpI Downloads

Each of the PowerOpI Framework distributions below focuses on one of the environments addressed by PowerOpI (Power Query, SQL Server, and MS Access) and they all solve the same two problems: Description, aggregation and explosion of a Program Management logistics problem using JOINs, and aggregation via UNION of multiple similar datasets. Data sources are based on Excel Structured Tables, and use INDEX/MATCH, SUMIF, XLOOKUP and related functions. Scheduling is done in MS Project and imported into the database solution. Visualization can be done using Visio, and examples are included in the distributions.

Within each, QUERYs for JOIN and UNION are demonstrated in several ways appropriate to its database environment (Direct Query, Stored Query/Stored Procedure, and in some cases using Linked Files, Linked Servers, or OPENROWSET) and using SQL, M, and M with embedded SQL. The examples demonstrate structures that have been shown to be necessary and sufficient in the Author's experience, to handle pretty large and complicated projects.

The methodologies can be extended to address different problems than illustrated; the concept of using Excel with a Database is very general and the tools are much more powerful when combined. Common commercial tools do not seem to be available to address the combined complexity of problems addressed. On the whole, commercial tools operate as "black holes", tied within specific domains and unable to address cross-functional requirements and practices. The answer is to combine tools specialized for domains, to aggregate their power.

The methodologies of each environment can be compared. Differences among the environments are compared in a table farther down the page; such differences generally affect ability to distribute and maintain solutions among multiple client users in an organizational cross-functional project, or to focus on single-user data mining and analysis.

PowerQueryTool

Free Functional Demo With Included Functional Templates. Shared Data Source - distributed Report Client.

  • PowerOpI Data Source for Single-Instance, and PowerOpI Distributed Client.
    • PowerOpI Support for PowerQuery (M-only in free version).
    • Logistics example Demo Project Data Source Tables With JOIN Keys.
    • PowerQuery Query & Connection manager.
    • PowerQuery M Formula Language QUERYs.
    • PowerQuery M-Formula Language Configured-Query editor, with Common Configuration Variables.
    • Logistics example Pivot Table Reports.
  • MS Project Task Data Import Stub.
  • For operation, also needs Framework for MS Project and MS Visio (separate download, below).

Download Page: PowerOpI PowerQueryTool

Consulting Available: Contact for Details

Framework for MS Project and MS Visio

Free Functional Demo With Included Functional Templates, for use with PowerQueryTool and PowerOpI Framework

  • PowerOpI Support for MS Project date input and MS Visio image output.
  • Project
    • MS Project Rollup with Calendared-Resource file and two inserted Subprojects.
    • MS Project Rollup Task Data Export file for use with MS Visio Image, and with PowerOpI Data Source.
  • Visio
    • MS Visio Image source, linked from Task Data Export file, and linked from PowerOpI Data Source file.
    • MS Visio Images Exported as .gif, .pdf, .png.

Download Page: PowerOpI Framework for MS Project and MS Visio

Consulting Available: Contact for Details

PowerOpI SQLServerTool

PowerOpI Framework for SQL Server with Demo Project. Shared Data Source - distributed Report Client.

  • Demo project based on Excel Structured Tables, Joined via SQL Server. Demo project illustrates Program Management Logistics project (program schedule, equipment use, structure, builds, expense and capital acquisition, and distribution), showing methodology extensible to organizational management topics such as Supply Chain, Financial Management, organizational structure and staff assignment.
  • Support for QUERY / JOIN using SQL Server via Data Connections (with SQL), and via PowerQuery (with M and SQL).
  • Sample SQL QUERYs and M QUERYs, with access to various file types and datatases, for use of direct QUERYs from Excel, Stored Procedures, Linked Servers, and OPENROWSET.
  • Sample SQL QUERYs, and M QUERYs with and without embedded SQL, and configured coherently using Common Configuration Variables.
  • Sample QUERYs for setup via SQL Server Management Studio (SSMS) / Visual Studio / SQL Server Data Tools (SSDT); configured coherently using Common Configuration Variables.

Download Page: PowerOpI SQLServerTool

Consulting Available: Contact for Details

PowerOpI MSAccessTool

PowerOpI Framework for MS Access with Demo Project. Standalone and Shared Data-Source - distributed Report Clients.

  • Demo project based on Excel Structured Tables, Joined via MSAccess. Demo project illustrates Program Management Logistics project (program schedule, equipment use, structure, builds, expense and capital acquisition, and distribution), showing methodology extensible to organizational management topics such as Supply Chain, Financial Management, organizational structure and staff assignment.
  • Support for QUERY / JOIN using MS Access via Data Connections (with SQL), and via PowerQuery (with M and SQL).
  • Sample SQL QUERYs and M QUERYs, with access to various file types and datatases, for use of direct QUERYs from Excel, and Stored QUERYs.
  • Sample SQL QUERYs, and M QUERYs with and without embedded SQL, and configured coherently using Common Configuration Variables.

Download Page: PowerOpI MSAccessTool

Consulting Available: Contact for Details

PowerOpI SQL Server Security

PowerOpI Framework for SQL Server security administration, targeted toward multi-user shared-server configurations.

  • Set up and manage Server Logins, Database Users, Permissions, Server Roles, Database Roles, Database Domains.
  • Driven from Tables in Excel that may be pushed into SQL Server.
  • Includes templates for Server Logins, Database Users, and Server and Database Roles for OpIClientRemote, OpIClientLocal, OpIDataOwner, OpIManager, and OpISuper.
  • Executed in T-SQL provided, to be run in SQL Server Management Studio.
  • Provided SQL can also be used a reference, illustrating: Use of SQL Variables, Cursors (Iterators), SQL Server system tables, complex Query syntax, WHILE constructs, IF EXISTS constructs, Temporary Tables, use of COUNT, SQL TRY/CATCH with error handling and reporting, Dynamic SQL creation using CONCAT; and more!

Download Page: PowerOpI SQL Server Security

Consulting Available: Contact for Details

PowerOpI MySQL Tool

PowerOpI Framework for MySQL.

  • Experimental, in development. Functional to connect to and query MySQL, but not fully developed with Program Management project template.
  • Supports ODBC, with Connection String or DSN.
  • Configured to connect to MySQL Community Server using Connector/ODBC with demo queries of included sakila schema.

Download Page: PowerOpIMySQLTool

Consulting Available: Contact for Details

PowerOpI Framework

All features of PowerOpI PowerQueryTool, plus

  • Framework with Demo Project for PowerOpI Standalone, Shared Data-Source, and Client
    • Power OpI support for PowerQuery (M and SQL); MS Access (SQL); SQL Server (SQL).
    • Sample SQL QUERYs, with configured-QUERY editor, with Common Configuration Variables.
    • Sample SQL QUERYs for use of Linked Servers, Stored Procedures, and Stored QUERYs, with sample QUERYs for setup via SQL Server Management Studio (SSMS).
  • For operation, also needs Framework for MS Project and MS Visio (separate download, above).

Download Page: PowerOpI Framework

Consulting Available: Contact for Details

The Framework is available in a free version that demonstrates and fully supports M with PowerQuery, and is also available in a consulting-required version that also supports SQL with PowerQuery, SQL Server, and MS Access for variations of distribution and maintenance characteristics. The structure is the same for both versions, with differences in provided Tables, QUERYs, and Connections.

FrameworkKey.png

Framework and Standalone The structure of the Framework is shown in the image just below. The Framework contains Reports, Tables, QUERYs, and Connections. In the PowerQuery version distributed, these objects are configured as a Standalone tool, containing both Data and Client objects. Report objects (Pivot Report, charts, and Tables) are driven by PowerQuery QUERYs, that pull data from Tables, via PowerQuery QUERYs. Table objects within the Framework workbook are directly accessed by PowerQuery. Another Table is provided in an external file accessed by a separate QUERY. PowerQuery itself is contained within the Excel environment; if SQL Server or MS Access were used, as in the Framework version supporting SQL, those database engines would be outside the Excel environment, to be accessed through Connection objects.

FrameworkBase.png

These objects can be configured differently as you create them. They would be created to represent your project and product structures, using structure mimicking the structures of the Framework objects. They could be created by cloning and modifying objects provided in the Framework distribution, or they could be directly created in the provided Connection and QUERY editors, or they could be created using native PowerQuery and Excel functionality.

The Framework objects can be created and connected in different configurations too. In the image above, the left-most Data Source could be an instance of a PowerOpI Framework configured as a Data Source, or any of the files or database types that can be connected to via PowerQuery, SQL Server, or MS Access. The image below shows how Data Source function can be separated from Client function, for distributed use. This can include use of multiple Data Sources interconnected with multiple Clients. Differences in characteristics of distribution among PowerQuery, SQL Server, and MS Access are compared in the table below.

FrameworkDeploy.png

Here is a depiction of top-level architecture implemented by the Framework. Other configurations derived from this are also supported, distributing one or more Data Sources with one or more Clients using a variety of database structures via MS Access, SQL Server, and PowerQuery.

ExcelDatabase.gif

Comparing MS Access, PowerQuery, and SQL Server, as used with PowerOpI

Which Database environment to use is determined by

Here's a comparison of those and related factors.

QUERY Engine Attribute Properties
MS Access Article Link(s)
QUERY Language
  • Graphical UI, generating SQL.
  • SQL (widely known, well documented).
QUERY Source Storage
  • Excel Client Data Connector.
  • MS Access Stored QUERY.
QUERY Execution Entity
  • MS Access Client Computer (Each Client runs MS Access).
Connection Path to Excel Data Source Tables
  • Client Excel Data Connection QUERY to MS Access QUERY to MS Access Linked Table to Excel Table Shadow Range, Named Range, or Worksheet.
Database Management Facility
  • MS Access application on Client or accessed from Client.
Distribution
  • Standalone local machine(s) or shared Data-Source(s) with distributed Client(s).
  • Client(s) and shared Data-Source(s) via \\Fileshare SMB, or RDT to shared system.
  • Components hosted on Windows (Native or VM).
Security
  • QUERYs hosted on client expose QUERY structure and file structure.
  • Stored QUERYs in MS Access shield QUERY structure and file structure.
Maintenance
  • Maintenance to QUERYs on each client machine.
  • Maintenance to QUERYs in Shared MS Access Stored QUERYs centralize maintenance work.
Data Sources
  • Excel Worksheets.
  • Excel Named Ranges.
  • Excel Shadow Ranges.
PowerQuery Article Link(s)
QUERY Language
  • M Function Language (poorly documented, quirky, powerful).
  • SQL embedded in M (SQL is widely known, well documented).
  • PowerQuery UI (functional but strangely quirky), generating M Function Language.
  • Take note that M language support for a Self-JOIN operation is very awkward. Self-JOIN operation is critical to exploding a BOM. You could explode about two layers of BOM with manual coding; but if you plan to explode more than that you should research that capability in M prior to committing your project to it. As a workaround, you could use M for your other project operations but then use M to execute SQL Self-JOINs within a SQL database environment (Stored Procedure in SQL Server, or MS Access or another) rather than native M within PowerQuery.
QUERY Source Storage
  • PowerQuery Client QUERY/Data Connector.
QUERY Execution Entity
  • PowerQuery Client computer (runs Excel or PowerBI Desktop), Parsed and “Folded” to server as possible.
Connection Path to Excel Data Source Tables
  • PowerQuery client M QUERY Connection to current or external Excel file Table (PowerQuery Native).
  • PowerQuery client M or SQL Embedded in M QUERY Connection to MS Access; or SQL Server Linked Server using Microsoft.ACE.OLEDB.XX, or OPENROWSET QUERY, or Stored Procedure using Linked Server or OPENROWSET; to Excel File to Table Shadow Range, Named Range, or Worksheet.
Database Management Facility
  • Each Client Excel PowerQuery User Interface.
  • SQL Server Management Studio (SSMS) for SQL embedded in M, using SQL Server.
  • MS Access client, for SQL embedded in M, using MS Access.
Distribution
  • Standalone local machine(s) or shared Data-Source(s) with distributed Client(s).
  • Client(s) and shared Data-Source(s) via \\Fileshare SMB, or RDT to shared system.
  • Components hosted on Windows (Native or VM).
Security
  • QUERYs hosted on client expose QUERY structure and file structure.
  • No Stored Procedures in shared data source to shield.
  • SQL QUERYs in MS Access stored QUERYs or SQL Server Stored Procedures, shield QUERY structure and file structure.
Maintenance
  • Maintenance to QUERYs on each client machine.
  • SQL QUERYs in MS Access stored QUERYs or SQL Server Stored Procedures, centralize maintenance.
Data Sources
  • Three dozen file and database types including OLEDB, ODBC Web, MySQL, Oracle, Azure, Amazon, SalesForce, NoSQL, PostgreSQL, DB2, XML, Text, Excel, …
SQL Server Article Link(s)
QUERY Language
  • SQL (widely known, well documented).
QUERY Source Storage
  • Excel Client Data Connector.
  • SQL Server Stored Procedure.
  • PowerQuery Client.
QUERY Execution Entity
  • SQL Server Host Computer.
Connection Path to Excel Data Source Tables
  • Client Excel Data Connection QUERY to SQL Server to Linked Server using Microsoft.ACE.OLEDB.XX to Excel Table Shadow Range, Named Range, or Worksheet.
  • Client Excel Data Connection QUERY to SQL Server OPENROWSET to Excel File to Table Shadow Range, Named Range, or Worksheet.
  • Client Excel Data Connection QUERY to SQL Server Stored Procedure to Linked Server as above or OPENROWSET as above, to Excel File to Table Shadow Range, Named Range, or Worksheet.
Database Management Facility
  • SQL Server Management Studio (SSMS) on Clients. Required for maintenance tasks for example to set up Shared Procedures, Linked Servers, Databases.
Distribution
  • Standalone local machine(s) or Shared SQL Server(s) with distributed Client(s).
  • Client(s) and Shared SQL Server(s) via \\Fileshare SMB, or RDT to shared system.
  • Server(s) hosted on Windows or Linux. Client(s) hosted on Windows (Native or VM).
  • Clients executing SQL Server stored procedures on shared SQL Server(s), can be hosted on Windows, MacOS (Native or VM), iOS, Android… .
Security
  • QUERYs hosted on client expose QUERY structure and file structure.
  • QUERYs in Stored Procedures in SQL Server shield QUERY structure and file structure.
  • Linked Servers in SQL Server shield file structure.
Maintenance
  • Maintenance to QUERYs on each client machine.
  • Maintenance to QUERY in Shared SQL Server Stored Procedures centralize maintenance work.
  • Maintenance to Shared SQL Server Linked Servers centralizes file directory maintenance.
Data Sources
  • OLEDB file types.
  • ODBC database types.
  • Azure SQL.

The Zen of Excel Pivot Tables

Most books covering Excel Pivot Tables pretty much focus on construction of Pivots. I use Pivots every day to organize and manage data critical to planning and operation in my function. That data changes constantly, new kinds of data are added, and new reporting is needed. I've found there is a level of design above the technical construction of the Pivot Table that helps make the Pivot Tables themselves more straightforward to create, and helps with better evolution and reporting. I think this embodies the Zen of Excel Pivot Tables.


Integrating Microsoft Project with Excel and Visio

ProjectLinks.gif

Microsoft Project is widely used by Program Managers to represent a project execution plan and to quantify a schedule. Project can output dates that can be integrated into a Logistics/Build Plan to tie together dates with quantities and configurations, and from that can drive a Finance/Budget plan with date-dependent records describing expenditures, suppliers, build-items and so forth.

Article: Project Links.


INDEX/MATCH and SUMIF with Excel Structured Tables

Excel Structured Tables allow Excel to provide revolutionary functionality like a database with PowerBI, or with a SQL database as described in Excel-Database, or to some degree among multiple tables within Excel.
Syntax for Structured References.
Structured Reference examples.
Absolute structured references.
Structured Reference using VBA, Excel Object Model, Application.WorksheetFunction, and constructed Structured Reference expressions.

Excel INDEX/MATCH/MATCH

Structured Reference addresses columns by their column name. Used with INDEX and MATCH functions or with the SUMIF function, data in tables can be addressed by content, essentially allowing Excel to find its own way around a worksheet.

Structured Reference can address multiple tables in a workbook by name, so data can be addressed by content throughout a workbook. The combination of functions INDEX and MATCH allows for lookup, and SUMIF provides aggregation of values among multiple records.

The INDEX function can be used as:

=INDEX( area, rowNum, columnNum )

specifying a cell by rowNum and columnNum within the table.

MATCH syntax is:

=MATCH( soughtValue, withinRange, Exact )

and returns the item number of a soughtValue within a range. Used with a Structured Table, the number returned could be the number of a row or column containing a soughtValue.

Putting INDEX and MATCH together:

=INDEX( tableName,
MATCH( rowItemSought, tableName[inColumnName], 0 ),
MATCH( columnNameSought, tableName[#Headers], 0 )
)


Within a table specified by the tableName, a row is sought having the value of the variable rowItemSought in the column named the value of variable inColumnName, and the value in that row is returned from the column whose name is the value of the variable columnNameSought in the #header row of the table.

The 0 Exact value specifies "exact" match; choices include ≤, =, and ≥, specified by -1, 0, and +1. This combination of functions behaves much like VLOOKUP or HLOOKUP, except that this lookup can be up, down, left and right, and column and row order don't matter.

Excel SUMIF

INDEX/MATCH can return a single value, like the serial number of "R730 Server" for system "HW-Qual". To return the total number of Ethernet cables associated with all RNICs distributed among several servers requires a function that can find all records for "RNIC"s and add up the number of "Ports" in those records. (Start thinking of table rows as Database Records). That function is SUMIF.

SUMIF is used as:

=SUMIF ( sourceTable[keyColumn],
keyValue,
sourceTable[valueColumn]
)

The sum of the values in column "valueColumn" for all records in table "sourceTable" containing the value of variable "keyValue" in column "keyColumn" is returned.

Other functions work similarly with Structured Tables and Structured References. In this type of application, some more that are particularly useful include SUMIFS, COUNTIF, COUNTIFS, MAXIFS, and IFS which allow you to sum, to count, and to choose based on one or multiple criteria, using Structured Table fields as arguments. They function just as the description of SUMIF just above; type in the table name followed by a "[" and Intellisense will drop down a list of field names to choose.

This article shows use of INDEX/MATCH and SUMIF with Structured Reference, in the Excel user interface. Those constructs can also be used from within VBA as described in Structured Reference using VBA, Excel Object Model, Application.WorksheetFunction, and constructed Structured Reference expressions.

The article linked just above also describes use of Excel function TEXTJOIN, which is useful for constructing a SQL string (see page Link Excel with SQL Server). TEXTJOIN is available as of Excel 2016 and is now available in VBA as a WORKSHEETFUNCTION. A function called VBTEXTJOIN is described in the linked article, that provides TEXTJOIN function from within VBA, is usable in the Excel user interface, and available in Excel current and prior editions.


XLOOKUP Used With Excel Structured Tables, Compared With INDEX/MATCH/MATCH

Excel XLOOKUP

(2019 August 30, updated 4 November and 2020 April 18) Microsoft introduced function XLOOKUP as a functional enhancement to VLOOKUP, announcement on Microsoft TechCommunity, and reference:

syntax:

itemRange = XLOOKUP( itemValueSought, lookup_array, return_array, [if_not_found], [match_mode], [search_mode] )

Example:

For example, an XLOOKUP function used in a "destination" table, to which a single value is returned to each destination table record from a "source" table, illustrating use of XLOOKUP within Excel Structured Tables:

=XLOOKUP( [@Key], tblItemData[Key], tblItemData[Cost] )

XLOOKUP uses a value in destination table records column "Key" to look up item records within source table "tblItemData" with matching value in their column "Key", to return a value from that record in column "Cost".

lookup_array:

  • cellAddr1:cellAddrN (vertical or horizontal vector, or range to search for itemValueSought).
  • tableName[inColumnName] (Structured Table column to search for itemValueSought).

return_array:

  • Source array containing item values to be returned. "return-from_array" would be a more descriptive argument name
  • XLOOKUP returns reference to a Range of values from return_array, corresponding to matches with itemValueSought found in the lookup_array.
  • A single match-instance found in lookup_array results in a single-item Range returned by XLOOKUP.
  • If multiple matches are found in lookup_array, a Range containing multiple items is returned by XLOOKUP. The Microsoft examples show wrapping an XLOOKUP function by the SUM function to add up all items in the Range returned by XLOOKUP, which could include 1 or multiple items.See note also at ***.
  • Zero items returned results in an #NA error return, hence the if_not_found parameter, or you could wrap XLOOKUP in an IFERROR function.

if_not_found:

  • The if_not_found parameter specifies a TEXT value to be returned from XLOOKUP if itemValueSought is not found in lookup_array. The TEXT value returned can be cast to another data TYPE, using functions external to the XLOOKUP function, if needed.

match_mode:

  • -1: <=
  • 0: =
  • 1: >=
  • 2: wildcard (*, ?, ~)

search_mode:

  • 1: first-to-last (default)
  • -1: last-to-first
  • 2: binary in ascending sort
  • -2: binary in descending sort

XLOOKUP removes the column-order requirement inherent to VLOOKUP.

Excel INDEX/MATCH/MATCH Review

Since many users of VLOOKUP will consider moving to XLOOKUP, XLOOKUP will also be considered as an alternative to INDEX/MATCH/MATCH, which is considered by many to be "complicated".

That's not quite so simple a comparison for applications using INDEX/MATCH/MATCH in Excel Structured Tables, needing to do 2-dimensional lookups using Structured References to define function arguments.

The INDEX/MATCH/MATCH structure used in the Excel user interface with Structured Tables and Structured References, as described in the preceding section on this page, looks like:

=INDEX( tableName,
MATCH( rowItemSought, tableName[inColumnName], 0 ),
MATCH( columnNameSought, tableName[#Headers], 0 )
)

where tableName is the name of the source Table from which values are to be looked up (and which could be a separate Table, or the same destination Table initiating the lookup). inColumnName names the column/field to be searched for a row matching rowItemSought. rowItemSought and columnNameSought could be literal strings, references to [@fields] in the same record as the INDEX/MATCH/MATCH expression, string specifying the tableName #Headers name for the field/column containing the value to be returned, or reference to a cell or named value elsewhere in Excel. These two arguments address the desired data as values of its attributes rather than by a hard address thereby implementing location-independence.

The first MATCH function determines row index in a searched column/field matching the value of rowItemSought; the second MATCH determines the column index of the return column named as the value of columnNameSought. The outer INDEX function uses the row index and column index within a table named as the value of tableName, and returns the value of the specified Table entry.

Using XLOOKUP to achieve the same functionality as INDEX/MATCH/MATCH

The analogous XLOOKUP structure used in the Excel user interface with Structured Tables and Structured References as above looks like:

=XLOOKUP( [@rowItemSought], tableName[inColumnName],
XLOOKUP( [@columnNameSought],
tableName[#Headers],
tableName[#Data] )
)

using the same definitions for the variables named as described for INDEX/MATCH/MATCH and showing some as references to [@same-row_fields].

The inner XLOOKUP returns the return_column range for columnNameSought, from which values will be returned. The outer XLOOKUP determines the row containing a matching rowItemSought value within the source Table column named inColumnName, and completes the lookup into the return_column, returning the value from its row containing the matching rowItemSought value. (Examples in announcement documents used cell addressing, so my effort was to determine how to use the functions with Structured Tables and Structured Reference.)

Example of XLOOKUP used as INDEX/MATCH/MATCH:

For example, XLOOKUP implementation of functionality similar to that described for INDEX/MATCH/MATCH. In that construction, the second MATCH function determines the appropriate field from which to return the value. That field name can be specified by reference, and is not necessarily embedded into the construction. An analogous construction using XLOOKUP:

=XLOOKUP( [@Key], tblItemData[Key],
XLOOKUP( [@ItemSought],
tblItemData[#Headers],
tblItemData[#Data] )
)

Behavior of the ItemSought parameter:

This structure adds an internal XLOOKUP, similar in function to the second MATCH parameter in INDEX/MATCH/MATCH, to find source table field named as ItemSought from which to return values.

In this example, the Key parameter specifies source records to find in a Structured Table named tblItemData. The ItemSought parameter specifies, either by value or by reference, the name of the column from which values will be returned from those records in the source Table by this construction. The ItemSought parameter can be specified as:

  • A string value, specified like "Cost" including the quotes, naming the field in the Source Table.
  • Name of a field in destination records, specified like [@ItemSought], containing the name the field in the source table from which values will be returned.
  • A named worksheet cell, specified like ItemSought without quotes, containing the name of the field to be returned from the Source Table.
  • CANNOT be a worksheet cell reference, specified like L$1, which would typically be a destination table column heading whose value would also specify the source table field name from which to return values. In this way XLOOKUP functionality differs from INDEX/MATCH/MATCH.

My interest is in returning a single value from a Structured Table, to be used within a calculation in a field within a Structured Table. There are more, broader, uses of XLOOKUP that can return a more complex range as their result.

XLOOKUP Returns a Range, not a Value (!!!)

*** (Updated 12 June 2020) I don't see real advantages in mapping XLOOKUP onto INDEX/MATCH/MATCH. Of course it works for single-value return as shown, but I find INDEX/MATCH/MATCH more intuitive. However, the really big difference between them is that XLOOKUP returns a RANGE, and that opens up a different world of uses for XLOOKUP.

(Note that the values returned in the Range from XLOOKUP may be selected from non-contiguous cells in the source, that meet the selection criteria.)

You could wrap the XLOOKUP function inside a SUM function to summarize or aggregate values in the RANGE returned by XLOOKUP: That's effectively SUMIF capability. In fact, you could wrap it with any function taking a range argument (e.g. Max, Min, Average, some index/search function, ...); or in VBA, include it (using Application.WorksheetFunction) in any calculation you program. That gives you versatile "CalcIF" capability. The "IF" function is controlled by the itemValueSought and lookup_array arguments in any calculation you program. So the "CalcIF" functionality works in both the Excel UI and in VBA.

Microsoft have already provided SUMIF, AVERAGEIF, COUNTIF, and corresponding ...IFS functions. Those can be extended for complex selection to work on any non-contiguously-selected range of values by using a helper field construction, described just below. But there may be other functions for which that would be useful, and the conditional "CalcIF" functionality of XLOOKUP can also be useful to functions you provide in VBA.

As an example, I need to construct a SQLCommandText string in a a field in each record in a destination Excel Structured Table by concatenating text values from a specified column in a another source Structured Table. The name of the column to be concatenated is specified in each destination record, by column-name contained in a field in that destination record.

So the column-name field in each destination-table record specifies the name of a source-table column from which I want to concatenate all contained values back into the destination-table SQLCommandText field. I can't embed the column-name into the concatenation expression, because I want to enable my user to choose any SQL source table column-name, into each destination-record source-column-name field, without forcing them to update my function calls.

So here's what my function call looks like in the destination table, using TEXTJOIN to concatenate a column with XLOOKUP to give it the named column from the source table:

=TEXTJOIN(CHAR(32) & CHAR(10),TRUE,
XLOOKUP([@SQLName],tblSQL[#Headers],tblSQL[#Data]))

XLOOKUP returns a range matching the source-table column tblSQL[SQLName] whose name is specified in destination-table field @SQLName and matched in tblSQL[#Headers]. From that, function TEXTJOIN concatenates all values in the returned Range. By the way, it's good practice to wrap all this with an IFERROR function just in case... consider returning a null field "" or else "-" on error.

A note on helper fields for record selection: If you need control on which elements are selected to participate in the wrapper calculation, you can create a structured-key "helper field" (e.g., "[@field1] & ":" & [@field2] & ":" & String" etc.) to help XLOOKUP select Table elements as you wish.

Excel XMATCH

Microsoft also announced a function XMATCH as a functional enhancement to MATCH, reference:

syntax:

itemIndex = XMATCH( itemValueSought, lookup_array, [match_mode], [search_Mode] )

lookup_array:

  • cellAddr1:cellAddrN (vertical or horizontal vector, or range to search for itemValueSought)
  • tableName[inColumnName]
  • tablename[#thisRow]
  • tablename[#Headers]

match_mode:

  • -1: <=
  • 0: =
  • 1: >=
  • 2: wildcard (*, ?, ~)

search_mode:

  • 1: first-to-last (default)
  • -1: last-to-first
  • 2: binary in ascending sort
  • -2: binary in descending sort

Consider also that the construction INDEX/XMATCH/XMATCH provides new lookup capability, given the optional wildcards of match_mode, and search order options of search_mode in each instance of XMATCH.

Finally, Microsoft provides both XLOOKUP and XMATCH in Application.WorksheetFunction, making them directly callable in VBA.

Assessment: XLOOKUP/XMATCH vs. INDEX/MATCH/MATCH

It seems that using XLOOKUP in Structured Tables is more or less the same capability and complexity as using INDEX/MATCH/MATCH.

  • So if you are already familiar with VLOOKUP and use either VLOOKUP or XLOOKUP outside of Structured Tables, you may be more comfortable using the familiar structure of XLOOKUP within Structured Tables, with inner-XLOOKUP jazz added for 2-D references and a little more jazz to express the arguments as Structured References.
  • Or if you're comfortable with INDEX/MATCH/MATCH and primarily use that within Excel Structured References, there's little reason to move from that structure to use XLOOKUP.
  • The optional arguments in XLOOKUP providing wildcard support and search-direction control, and the ability to return a multiple-item range from XLOOKUP will add new lookup capabilities and new best-practices.

(Updated 12 June 2020) But the return of a Range by XLOOKUP is great new functionality, allowing operation on a selected set of contiguous or non-contiguous members. This function can be provided to other functions operating on a Range, provided by Microsoft or by your VBA and greatly leveraging the functionality of those functions.

What if your users are on an earlier version of Excel that doesn't include XLOOKUP?'

XLOOKUP is available in Excel AFTER Excel versions 2016 and 2019, so in some cases you can't count on availability of XLOOKUP. Here's a limited-function emulation of XLOOKUP to return a Column with a specified Name from a Structured Table.

Function vbatableXLOOKUP allows selection of a column to return by naming the column in a table field, rather than hard-coding the name in each function call in the table. This is so that the intended column can be chosen by drop-down, or calculation.


Excel LAMBDA and LET Functions

I needed to determine the third and fourth Wednesday of specified months, to synchronize invoicing with a government payment schedule. I looked for functions to do that and found some that partially worked but would only return the third Wednesday and fail to find other dates specified.

I decided to use a LAMBDA expression to do that. There are good references describing how to write a LAMBDA function; for example https://peltiertech.com/calculate-nice-axis-scales-with-let-and-lambda/.

LAMBDA function lets you create a function, with arguments specified in the call (as value, cell reference, named reference, Structured Table field reference); and return calculated value of an expression that uses those arguments The LAMBDA function is implemented from functions available on a worksheet, instead of VBA.

LET function lets you collect arguments (as values, cells, named references), name them, and name and return value of an expression that uses those arguments.

LET functions may be used within LAMBDA functions, referencing each other via arguments and named value returns, to perform sequential work to be used in the value returned by the LAMBDA function.

Here's an example LAMBDA, to find the Nth specified weekday in a specified month and year; it can also find in reverse from the end of a month. So examples include fourth Thursday in November, last or next to last Friday in a month.

Following that are some helper functions that convert back and forth between day and month name and abbreviations; and month number in year, day number in week.

fnTargetInstanceDate

=LAMBDA(tgtMonth,tgtYear,WkdayInstance,WkdayTarget,
    IF(WkdayInstance > 0,
        LET(
                IndexDate, DATE(tgtYear, fnMonthNameToNumber(tgtMonth), 8),
                IndexDate - WEEKDAY(IndexDate - fnDayNameToNumber(WkdayTarget))
               +(WkdayInstance-1)*7
         ),
           IF(WkdayInstance < 0,
               LET(
                       IndexDate, DATE(tgtYear, fnMonthNameToNumber(tgtMonth)+1, 1),
                       IndexDate - WEEKDAY(IndexDate - fnDayNameToNumber(WkdayTarget))
                      +(WkdayInstance+1)*7
                ), 0
           )
    )
 )
([@PaymentMonth],[@PaymentYear],[@TgtWkdayInstance],[@WeekdayTargetName])

This function can be specified in the Excel Name Manager as a named value which will execute the function when called.

=LAMBDA(tgtMonth,tgtYear,WkdayInstance,WkdayTarget, IF(WkdayInstance > 0, LET( IndexDate, DATE(tgtYear, fnToMonthNumber(tgtMonth), 8), IndexDate - WEEKDAY(IndexDate - fnToDayNumber(WkdayTarget)) +(WkdayInstance-1)*7 ), IF(WkdayInstance < 0, LET( IndexDate, DATE(tgtYear, fnToMonthNumber(tgtMonth)+1, 1), IndexDate - WEEKDAY(IndexDate - fnToDayNumber(WkdayTarget)) +(WkdayInstance+1)*7 ), 0 ) ) )

NOTE no argument string at end of expression above. Need argument string to execute in cell; Drop argument string for named expression.

Here's how to call it:
=fnTargetInstanceDate([@PaymentMonth],[@PaymentYear],[@TgtWkdayInstance],[@WeekdayTargetName])
Example using Structured Table Reference

The function uses helper functions, fnToDayNumber and fnToMonthNumber, which are also LAMBDA functions and are specified below with similar functions to give a complete set of conversions. In general, each can be called with day names or month names as appropriate, such as "September", "Sept", "Sep", "9" or "Tuesday", "Tues", "Tue", "3". The functions return strings. They also return identity values such as a call to long version of "September" returns "September"; this is so that the functions can be used without knowing whether a conversion is needed or not; and accepting the argument in several forms anyway.

Explanation of the function:
An index date is determined, and is used to help determine the first instance of the specified weekday in the month. Multiples of week length = 7 are then added to determine the Nth occurance of that weekday. The index date chosen is the 8th of the month, because the first instance of any weekday in a month will be contained in the first 7 days. Counting backwards from the 8th assures that the closest preceding instance of a weekday is in the same month as the 8th of the target month. The first is a solid index date in general for date calculations, and moving the index to the first + 7 simplifies the math somewhat.

The calculation is slightly modified to count back from the end of the month. The last day varies from month-to-month (28, 29, 30, 31) but can be determined as 1 day prior to the first of the NEXT month, a reliable calculation. The last instance of a targeted weekday will occur in the 7 days prior to the first of the next month; and multiples of 7 can be subtracted to find the target weekday in prior weeks. The function counts backward from the end of month if the argument is negative; -1 indicates the last instance of a specified weekday, -2 is the week prior to the last, etc.

Here are month and day conversion functions that are useful in date conversions.

fnMonthNameToLong

=LAMBDA(Argument,
    LET(
        MonthNameToLong,Argument,
        IFERROR(
            SWITCH(
                IF(ISNUMBER(Argument),TEXT(Argument,"#0"),Argument), 
                "January", "January", "Jan", "January", "1", "January", "February", "February", 
                "Feb", "February", "2", "February", "March", "March", "Mar", "March", "3", "March", 
                "April", "April", "Apr", "April", "4", "April", "May", "May", "5", "May", "June", 
                "June", "Jun", "June", "6", "June", "July", "July", "Jul", "July", "7", "July", 
                "August", "August", "Aug", "August", "8", "August", "September", "September", 
                "Sep", "September", "Sept", "September", "9", "September", "October", "October", 
                "Oct", "October", "10", "October", "November", "November", "Nov", "November", "11", 
                "November", "December", "December", "Dec", "December", "12", "December"
            ), 
        "-")
    )
)($I$28)

NOTE argument string at end of expression above. Need argument string to execute in cell; Drop argument string for named expression.

fnToMonthNameLong
=LAMBDA(Argument,LET(MonthNameToLong,Argument,IFERROR(SWITCH(IF(ISNUMBER(Argument),TEXT(Argument,"#0"),
Argument), "January", "January", "Jan", "January", "1", "January", "February", "February", "Feb", "February", "2", "February", "March", "March", "Mar", "March", "3", "March", "April", "April", "Apr", "April", "4", "April", "May", "May", "5", "May", "June", "June", "Jun", "June", "6", "June", "July", "July", "Jul", "July", "7", "July", "August", "August", "Aug", "August", "8", "August", "September", "September", "Sep", "September", "Sept", "September", "9", "September", "October", "October", "Oct", "October", "10", "October", "November", "November", "Nov", "November", "11", "November", "December", "December", "Dec", "December", "12", "December"), "-")))($I$17)

NOTE argument string at end of expression above. Need argument string to execute in cell; Drop argument string for named expression.

=fnToMonthNameLong($I$17)

fnToMonthNameShort
=LAMBDA(Argument,LET(MonthNameToShort,Argument,
IFERROR(SWITCH(IF(ISNUMBER(Argument),TEXT(Argument,"#0"),Argument), "January", "Jan", "Jan", "Jan", "1", "Jan", "February", "Feb", "Feb", "Feb", "2", "Feb", "March", "Mar", "Mar", "Mar", "3", "Mar", "April", "Apr", "Apr", "Apr", "4", "Apr", "May", "May", "5", "May", "June", "Jun", "Jun", "Jun", "6", "Jun", "July", "Jul", "Jul", "Jul", "7", "Jul", "August", "Aug", "Aug", "Aug", "8", "Aug", "September", "Sep", "Sep", "Sep", "Sept", "Sep", "9", "Sep", "October", "Oct", "Oct", "Oct", "10", "Oct", "November", "Nov", "Nov", "Nov", "11", "Nov", "December", "Dec", "Dec", "Dec", "12", "Dec"), "-")))($I$17)

NOTE argument string at end of expression above. Need argument string to execute in cell; Drop argument string for named expression.

=fnToMonthNameShort($I$17)

fnToMonthNumber
=LAMBDA(Argument,LET(MonthNameToNumber,Argument,
IFERROR(VALUE(SWITCH(IF(ISNUMBER(Argument),TEXT(Argument,"#0"),Argument), "January", "1", "Jan", "1", "1", "1", "February", "2", "Feb", "2", "2", "2", "March", "3", "Mar", "3", "3", "3", "April", "4", "Apr", "4", "4", "4", "May", "5", "5", "5", "June", "6", "Jun", "6", "6", "6", "July", "7", "Jul", "7", "7", "7", "August", "8", "Aug", "8", "8", "8", "September", "9", "Sep", "9", "Sept", "9", "9", "9", "October", "10", "Oct", "10", "10", "10", "November", "11", "Nov", "11", "11", "11", "December", "12", "Dec", "12", "12", "12")), 0)))($I$17)

NOTE argument string at end of expression above. Need argument string to execute in cell; Drop argument string for named expression.

=fnToMonthNumber($I$17)

fnToDayNameLong
=LAMBDA(Argument,LET(DayNameToLong,Argument,
IFERROR(SWITCH(IF(ISNUMBER(Argument),TEXT(Argument,"#0"),Argument), "Sunday", "Sunday", "Sun", "Sunday", "1", "Sunday", "Monday", "Monday", "Mon", "Monday", "2", "Monday", "Tuesday", "Tuesday", "Tue", "Tuesday", "Tues", "Tuesday", "3", "Tuesday", "Wednesday", "Wednesday", "Wed", "Wednesday", "Weds", "Wednesday", "4", "Wednesday", "Thursday", "Thursday", "Thu", "Thursday", "Thurs", "Thursday", "5", "Thursday", "Friday", "Friday", "Fri", "Friday", "6", "Friday", "Saturday", "Saturday", "Sat", "Saturday", "7", "Saturday"), "-")))($I$33)

NOTE argument string at end of expression above. Need argument string to execute in cell; Drop argument string for named expression.

=fnToDayNameLong($I$33)

fnToDayNameShort
=LAMBDA(Argument,LET(DayNameToShort,Argument,
IFERROR(SWITCH(IF(ISNUMBER(Argument),TEXT(Argument,"#0"),Argument), "Sunday", "Sun", "Sun", "Sun", "1", "Sun", "Monday", "Mon", "Mon", "Mon", "2", "Mon", "Tuesday", "Tue", "Tue", "Tue", "Tues", "Tue", "3", "Tue", "Wednesday", "Wed", "Wed", "Wed", "Weds", "Wed", "4", "Wed", "Thursday", "Thu", "Thu", "Thu", "Thurs", "Thu", "5", "Thu", "Friday", "Fri", "Fri", "Fri", "6", "Fri", "Saturday", "Sat", "Sat", "Sat", "7", "Sat"), "-")))($I$33)

NOTE argument string at end of expression above. Need argument string to execute in cell; Drop argument string for named expression.

=fnToDayNameShort($I$33)

fnToDayNumber
=LAMBDA(Argument,LET(DayNameToNumber,Argument,
IFERROR(VALUE(SWITCH(IF(ISNUMBER(Argument),TEXT(Argument,"#0"),Argument), "Sunday", "1", "Sun", "1", "1", "1", "Monday", "2", "Mon", "2", "2", "2", "Tuesday", "3", "Tue", "3", "Tues", "3", "3", "3", "Wednesday", "4", "Wed", "4", "Weds", "4", "4", "4", "Thursday", "5", "Thu", "5", "Thurs", "5", "5", "5", "Friday", "6", "Fri", "6", "6", "6", "Saturday", "7", "Sat", "7", "7", "7")), 0)))($I$33)

NOTE argument string at end of expression above. Need argument string to execute in cell; Drop argument string for named expression.

=fnToDayNumber($I$33)


Data Visualization

When linked as shown on the projectlinks page, a Microsoft Visio image will illustrate the MS Project plan.

SystemTop1175

Now you have the skills to put together really powerful and customizable tools to manage project logistics and financials, using Excel with a database to use JOINs linking Excel tables with data from other programs. As the database develops and changes to reflect your execution history and plan adjustments, you will find that you can use data-mining to get information to better manage.

This comes in the form of well-designed pivots to organize and summarize data; SQL SELECT and JOIN to combine related datasets and perform complicated data selection and organization. And Data Visualization which will help you understand the data as you develop management structure, and to explain the data-driven management structure to others.

PieChart

On Data Visualization, I gave a reference to Tufte below, a key figure in Data Visualization. Also see Excelcharts.com by Jorge Camoes, and John Walkenbach's The Spreadsheet Page. Stephen Few is a key mentor in data visualization and runs Perceptual Edge, a blog focusing on Visual Business Intelligence.

My charts haven't appeared yet on Kaiser Fung's Junkcharts blog. But you should go there, to see many examples and discussion of charts that do or don't work. I'm sure I will get there, though, because I have certainly used pie charts, which are not popular among Visualization experts. Just make sure your analysis remains faithful to the data and that the data, rather than your presentation method, remains the story. And even pie charts have their use, depending on the point you are making. Pies are good at illustrating relative proportions of component parts within a whole collection of parts.

You could compare variables "sliced" within separate pies by scaling the areas of each of the pies. Knowing that two pies would have areas a = π * r2 and A = π * R2, you would make the area A of one pie comparable to area a of the other pie and scaled by k such that A = k * a, by setting R = r *  k .

Having said all that, and for proper use only, here is the One Pie To Rule All Pies. This Pie Chart is done in Excel, but you won't find it in the menus. Nobody else will have these graphics, and the brilliance of your analysis will pop right off the page! Article: Pie Chart.


Connected Chart Labels

You can put a label on an Excel chart tied to data and computation on a spreadsheet. For example, you can reference assumptions, or a total that may not be displayed explicitly in the chart data such as a total on a pie chart. Article: Connected Chart Labels.

Using Excel Tables in VBA

Results of a database QUERY may be returned to an Excel Table to be processed by VBA code running in Excel. Jan Karel Pieterse has a very informative article on his JKP Application Development Services site describing code for using Excel Tables, followed by useful Q&A. See The Spreadsheet Page.


Programming References

Microsoft Excel

Excel 2007 VBAExcel 2007 VBA
Programmer's Reference

John Green, Stephen Bullen, Rob Bovey, Michael Alexander
Copyright © 2007 Wiley Publishing, Inc.
Published by Wrox an imprint of Wiley Publishing, Inc.

Best programming reference I've found. In depth, advanced. Focused on VBA programming.

Professional Excel DevelopmentProfessional Excel Development Second Edition
Rob Bovey, Dennis Wallentin, Stephen Bullen, John Green
Copyright © 2009 Pearson Education, Inc.
Published by Addison-Wesley

Definitive on design of Excel applications including VBA and Excel constructs. Best description of many useful techniques such Dynamic Lists.

Collie Power Pivot and Power BI

Copyright © 2016, Robert Collie and Tickling Keys, Inc.
Rob Collie & Avi Singh
Published by Holy Macro! Books

Go-To book describing use of DAX, PowerQuery, Power BI and Power Pivot in Excel 2010 - 2016.

PowerPivot Microsoft Business Intelligence Tools for Excel Analysts

Copyright © 2014, John Wiley & Sons, Inc.
Michael Alexander, Jared Decker, Bernard Wehbe
Published by John Wiley & Sons, Inc.

Covers Excel Business Intelligence basics, and in particular covers interaction between Excel and SQL Server.

Microsoft Excel 2019 Pivot Table Data CrunchingMicrosoft Excel 2019 Pivot Table Data Crunching

Bill Jelen and Michael Alexander
Copyright © 2019 Pearson Education, Inc.
Published by Microsoft Press

Pivot tables and VB are the most powerful features of Excel... Dives into PowerPivot and BI as well as classic Pivot Tables.

Mr. Excel Mr. Excel.

LinkedIn Microsoft Excel Users LinkedIn Microsoft Excel Users .

LinkedIn Excel Developers LinkedIn Excel Developers .

XML

XML Application Development with MSXML 4.0XML Application Development with MSXML 4.0
Stephen Mohr, Steven Livingstone, Darshan Singh, Danny Ayres, Michael Coming
Copyright © 2002 Wrox Press, Inc.

Old, and XML is now implemented in .NET, but this is a good overall reference and much translates to use in .NET.


Microsoft Project

Microsoft Office Project 2007 Inside OutMicrosoft Office Project 2007 Inside Out
Teresa S. Stover
Copyright © 2004 Teresa Stover
Published by Microsoft Press



Excel-based Gantt by Craig Hatmaker.

Data Visualization

Visual Display of Quantitative DataThe Visual Display of Quantitative Data
Edward R. Tufte
Copyright © 2001 Graphics Press


The point of a quantitative graphic: the revelation of the complex.

Charting complex information can be an art form, quantitatively illustrating regular structure of diverse phenomena. This is what shows true understanding, much more powerful than just illustration or enumeration.

Also see Excelcharts.com for a blog and tutorial on Excel data visualization using charts and dashboards. Tufte meets Excel, as they say on the site.

Monte Carlo Simulation

It is frequently useful to build a model using stochastic variables, and to use Monte Carlo simulation to create and analyze the range of probable outcomes. Following are add-ons for use with Excel and Project.

Oracle Crystal BallOracle Crystal Ball for Excel

Crystal Ball was originally developed by Decisioneering.

Palisade @RISK Palisade @RISK for Excel and Project

Palisade provides several more analysis tools for Excel including StatTools, Evolver, and PrecisionTree.

More Links

Microsoft Access

Access is well-integrated into MS Office and is fully-capable for many user database applications. Includes full UI support and VB. You can use SQL, but for many applications it is not needed. Access is a member of Microsoft Office.

Access 2007 Inside OutAccess 2007 Inside Out
John L. Viescas, Jeff Conrad
Copyright © 2007 Microsoft Press





Access 2003 Inside TrackAccess 2003 Inside Track
John Pierce, Paul Pardi
Copyright © 2004 Microsoft Press


This one is a non-intimidating introduction.

Excel and Access IntegrationMicrosoft Excel and Access Integration: With Microsoft Office 2007
Michael Alexander and Geoffrey Clark
Copyright © 2007 Wiley Publishing, Inc.




QueryStorm

A nice way to use SQL and C# from within Excel. Professional and Runtime licenses full-featured for commercial use. Free license available for non-commercial use only with local database.
QueryStorm

Microsoft SQL Server

SQL Server is a full-function database. It can be integrated with Excel, or most frequently used with a programmed client such as a .NET program. You need SQL for this. SQL Server is a separate product, but SQL Server Express is free.

SQL Server 2019SQL Server 2019 for developers
Bryan Syverson, Joel Murach
Copyright © 2020 Mike Murach & Associates, Inc.

Broad description of programming use of SQL Server 2019.

This is an excellent Go-To book on setting up and using the primary characteristics of SQL Server including queries and joins; data update, types and functions; database structure design and setup using SQL; views, scripts and stored procedures; transactions; security logins, permissions and roles; XML; and blobs.


Beginning T-SQLBeginning T-SQL with Microsoft SQL Server 2005 and 2008
Paul Turley, Dan Wood
Copyright © 2009 Wiley Publishing, Inc.

T-SQL, an extension of ANSI SQL implemented in SQL Server but not in Access, adds significant power to SQL Server.


Microsoft SQL Server 2008 ProgrammingMicrosoft SQL Server 2008 Programming
Robert Vieira
Copyright © 2009 Wiley Publishing, Inc.
Published by Wrox an imprint of Wiley Publishing, Inc.



ADO.NET

ADO.NET provides services for programmed connection between a database (Access, SQL Server, or other) and a client (Excel, or a .NET client program).

Murach's ADO.NET 3.5Murach's ADO.NET 3.5, LINQ and the Entity Framework with VB 2008
Anne Boehm
Copyright © 2009 Mike Murach & Associates, Inc.




Programming Languages: C/C++

A little off-topic, but as long as we're recommending books, here are two books on C/C++ for hardcore programmers. I have read quite a few books on C/C++ and these were the best by far. Read the books, stick with it, do the problems, and you WILL know how to use the languages. These are later editions than I used.

Johnsonbaugh and Kalin Applications Programming in ANSI CApplications Programming in ANSI C
Richard Johnsonbaugh and Martin Kalin
Copyright © 1996 Prentice Hall





Object-Oriented Programming in C++Object-Oriented Programming in C++
Richard Johnsonbaugh and Martin Kalin
Copyright © 2007 Pearson Custom Publishing; Custom Editions for Northeastern University




Design Patterns

Standing on the shoulders of those who have been there before you:

Erich Gamma - Design Patterns: Elements of Reusable Object-Oriented SoftwareDesign Patterns: Elements of Reusable Object-Oriented Software
Erich Gamma, Richard Helm, Ralph Johnson, John Vlissides
Copyright © 1994 Addison-Wesley Professional






Web Design Environments

Development Tools and Client Side

  • References and Tutorials
  • Microsoft Visual Studio
    • Microsoft Visual Studio is a comprehensive development environment: all languages, client-side, server-side, database. HTML, CSS, PHP, LESS, CoffeeScript, Python, Django, T-SQL, VB, C/C++/C#, F#; Javascript, JSON, XML/XAML etc. Huge ecosystem of plug-ins and tools supporting just about every programming language/framework/support/tool. VS Web Tools
    • Visual Studio Community Edition is free for personal use. It is full-function current-edition Visual Studio.
    • Visual Studio supports plug-in tools for development of PHP, Python, SQL and more, including syntax coloring, Intellisense, debug etc. (Community edition licensing may not allow some plug-in extensions, however.)
    • VSCode (Visual Studio Code), free, from Microsoft, a popular editor with built-in support and many add-ons available for web libraries, frameworks, support, and tools.
    • To display code nicely: Website-based-formatting Google code-prettify for C-like and HTML-like languages and more. Source-formatting editor-based Prettier for Javascript-like libraries, frameworks and support including Javascript, TypeScript, JSON, HTML5, Vue, Angular, Less, GrafQL, YAML and more, with plan to add Java, PHP, PostgreSQL, Python, Ruby, Swift and even more. Need a prettifier for React, don't we?
  • Adobe Dreamweaver
    • Adobe Dreamweaver is a leading website designer, focused toward creative design of client-side.
  • Wordpress (free)
    • Wordpress is leading example of a website Content Management System. Wordpress.org provides tutorials, software, plug-ins, themes etc.
    • Many add-on themes and plug-ins are available for Wordpress, both free and commercial. Many are listed at Wordpress.org. ThinkupThemes is a good example of a provider of commercial Wordpress site themes.
    • Wordpress.com provides hosting for Wordpress sites/blogs. You can also host a Wordpress client site on most ISP or cloud host sites.
    • Other popular CMSs (See extensive list) include Drupal, Joomla!, and WiX.These systems are based on a wide variety of open source and proprietary infrastructures including PHP, HTML 5, Python, Java, Perl, ASP, Ruby on Rails and others. Most CMSs support MySQL and some also support MS SQL Server.
  • Javascript
  • Graphics Tools
  • Perforce (free for up to 20 users)

Server-Side Environments

  • Hosting Alternatives
    • Hosting alternatives include: Cloud services (Microsoft Azure, Amazon Web Services), Apache Web server (Open source, free), ISP Website Hosting.
    • A production website would probably be hosted on a corporate site, a cloud provider or an ISP Website Host (which may be based on Apache or IIS). A developer can also install and use IIS or Apache as a local test server for site development.
    • Microsoft IIS (free) is included in Windows 8. How to install and configure IIS. (Download link).
      • You will probably set up your website on your computer in a directory different than the default set up when IIS is installed. To do that, you set up a Virtual Directory under the Default Web Site in IIS. When doing so, set up Read or Read and Execute permission for users "IIS AppPool\DefaultAppPool", and "IUSR". Here is a reference.
  • PHP (Open source, free)
    • Server-side programming for dynamic websites, form response, and database access. Straightforward, fully functional, easy to learn and use.
    • PHP.org site: documentation and download.
    • If using IIS, Download and install PHP using Web Platform Installer within IIS.
    • PHP and MySQL for Dynamic Web SitesPHP and MySQL for Dynamic Web Sites, Fourth Edition
      Larry Ullman
      Copyright © 2012 Larry Ullman
      Published by Peachpit Press




    • XDebug PHP debugger.
    • PHP Tools for MS Visual Studio: syntax coloring, Intellisense, PHP edit in HTML, supports XDebug, et al. (VS licensing for Express/Community edition does not support this plug-in.)
    • VS.php PHP development environment for MS Visual Studio.
  • Python and Django (Open source, free)
    • Server-side object-oriented programming language for dynamic websites, form response, and database access. There is a tutorial on the Python.org site.
    • Python download link.
    • If using IIS, Download and install Python using Web Platform Installer within IIS.
      • Configuring Python on IIS.
      • When specifying directory paths when configuring Python in IIS, make sure to use DOS 8.3 names. Fully spelled-out Windows names will prevent functionality.
      • Security has to be set for User DefaultAppPool on the Website and on the Python App. In Permissions add DefaultAppPool as User "IIS AppPool\DefaultAppPool" and set to Read/Exec, List, Read. Also set for IUSR and IIS_IUSRS.
    • Python Tools for Visual Studio
    • Django (Open source, free)

Database