From 6e0231074111ff1fda0b2de1765e1988fda2e9c9 Mon Sep 17 00:00:00 2001 From: Brendan Cunningham Date: Fri, 7 May 2021 16:33:35 -0400 Subject: [PATCH] README: add section about OMPI 4.1.x, OFI BTL, and high-PPN jobs. Signed-off-by: Brendan Cunningham --- README | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/README b/README index 7990555..2961534 100644 --- a/README +++ b/README @@ -67,7 +67,6 @@ Contains the following sections: - INSTALLING * INSTALLING USING MAKEFILE * INSTALLING USING EITHER YUM OR DNF -- TESTING - RELATED SOFTWARE TO PSM2 - SUPPORTING DOCUMENTATION @@ -251,6 +250,24 @@ libraries available on them. Open MPI provides a standard configure, make and make install mechanism which will detect and build the relevant PSM2 network modules for Open MPI once the header and runtime files are detected. +Open MPI 4.1.x, OFI BTL, and high PPN jobs +---------------- +Open MPI added the OFI BTL for one-sided communication. On an OPA fabric, the +OFI BTL may use the PSM2 OFI provider underneath. If PSM2 is in-use as both +the MTL (directly or via OFI) and the BTL (via OFI), then each rank in the +Open MPI job will require two PSM2 endpoints and PSM2 context-sharing will +be disabled. + +In this case, total number of PSM2 ranks on a node can be no more than: + (num_hfi * num_user_contexts)/2 +Where num_user_contexts is typically equal to the number of physical CPU +cores on that node. + +If your job does not require an inter-node BTL (e.g. OFI), then you can +disable the OFI BTL in one of two ways: + 1. When building Open MPI, specify '--with-ofi=no' when you run 'configure'. + 2. When running your Open MPI job, add '-mca btl self,vader'. + MVAPICH2 support ---------------- MVAPICH2 supports PSM2 transport for optimized communication on HFI hardware. -- 2.32.0