Set Regularities and Feasibility Problems
Alexander Y. Kruger, D. Russell Luke, Nguyen H. Thao
Introduction
In recent years there has been a tremendous interest in first-order methods for solving variational problems. As the name suggests, these methods only use information that, in some way, encodes the gradient of a function to be minimized. Often one has in mind the following universal optimization problem for such methods
All of the (nonconvex) convergence results concerning local linear convergence that we have seen in the literature involve implicitly or explicitly assumptions on the regularity of the functions and on the relation of the functions to each other at critical points. Prominent examples of this are the assumption that the functions satisfy the Kurdyka-Łojasiewicz property Loj63; Kur98; BolDanLew06; BolDanLeyMaz10 or that certain constraint qualifications are satisfied at critical points, like interiority of the constraints and invertability of the Jacobian of the objective in directions normal to the constraints.
Our goal in this note is to identify common ideas and properties for a special case of (1) and to develop a general framework that both encapsulates all of these ideas and is robust enough to be applied in other settings. We focus our attention on the feasibility problem
which is the specialization of (1) to the case
In the setting of the feasibility problem, regularity properties of individual sets (elemental regularity – see Section 3) and of their intersections (transversality and subtransversality of collections of sets – see Section 4) come into play. Both types of regularity have long history. A typical classical elemental regularity assumption is the convexity of the sets, while the traditional assumption that sets have a point in common on their interiors provides an example of transversal regularity in the convex setting. Another classical example of the second type of regularity is the concept of transversality of smooth manifolds.
In the last decade there has been a great deal of interest in extending the classical notions of regularity to include nonconvex and nonsmooth sets, motivated to a large extent by nonsmooth and nonconvex optimization and attendant subdifferential and coderivative calculus, optimality and stationarity conditions and convergence analysis of algorithms. Examples of modern elemental regularity concepts include Clarke regularity RocWet98, prox-regularity (Poliquin, Rockafellar, and Thibault PolRocThi00), super-regularity (Lewis, Luke, and Malick LewLukMal09), -regularity and -regularity (Bauschke, Luke, Phan, and Wang BauLukPhaWan13.2), relative ()-subregularity (Hesse and Luke HesLuk13), relative -Hölder regularity (Noll and Rondepierre NolRon15). Among the numerous transversal regularity (regularity of intersections) concepts we mention Jameson properties (N) and (G) Jam72), the conical hull intersection property (CHIP) (Chui, Deutsch, and Ward ChuDeuWar90), (local, bounded) linear regularity (Bauschke and Borwein BauBor93 and Zheng and Ng ZheNg08), the strong conical hull intersection property (strong CHIP) (Deutsch, Li, and Ward DeuLiWar97), metric regularity (Li Li97), metric inequality (Ngai and Théra NgaThe01), the closed intersection property, the normal property, the weak, uniform and dual normal properties, the normal conical hull intersection property (normal CHIP) (Bakan, Deutsch, and Li BakDeuLi05), the normal qualification condition (Mordukhovich Mor06), regularity, strong regularity, or uniform regularity (Kruger Kru05; Kru06; Kru09), the linearly regular intersection (Lewis, Luke, and Malick LewLukMal09), linear coherence, alliedness (Penot Pen13), the -qualification condition (Bauschke, Luke, Phan, and Wang BauLukPhaWan13.2), inherent and intrinsic transversality (Drusvyatskiy, Ioffe, and Lewis DruIofLew14; DruIofLew15), separable intersection (Noll and Rondepierre NolRon15), transversality and subtransversality (Ioffe Iof15). Some of the above concepts used under different names by different authors actually coincide.
A short survey of the recent developments in this area in the general nonsmooth and nonconvex setting with the emphasis on convergence analysis is provided in Section 2. The elemental and transversal regularity properties are further studied in Sections 3 and 4, respectively.
Section 3 introduces a general framework for elemental regularity of sets that provides a common language for the many different definitions that have appeared to date. This new framework makes the cascade of implications between the different types of regularity more transparent, namely that convexity prox-regularity super-regularity Clarke regularity -regularity -subregularity -Hölder regularity see Theorem 4. The last of these implications is new.
Section 4 focuses on two local regularity properties of collections of sets which we call here subtransversality and transversality. Both properties admit several representations and characterizations: metric, dual, angle, etc, and because of that or just historically are known under various names, e.g. (local) linear regularity, metric regularity, linear coherence, and metric inequality for the first property, and strong regularity, uniform regularity, alliedness, normal qualification condition for the second one. They correspond (are in a sense equivalent) to subregularity and metric regularity of set-valued mappings, respectively. At the same time, the properties are related to certain ‘good’ mutual arrangement of several objects (sets) in space, and after a discussion with experts in variational analysisWe are particulary indebted to Alex Ioffe for thoughtful and persuasive discussions., we have decided to adopt the classical ‘transversality’ terminology.
For ease of exposition, our discussion is limited to the case of just two closed subsets of Euclidean space with nonempty intersection, though most of the regularity properties discussed are easily extended to collections of more than two sets with nonempty intersection. We compare the various representations of these properties and discuss also some recently introduced ‘restricted’ regularity properties. A number of characterizations of the properties are formulated. Some characterizations are new, see Theorem 4.1(iii), Theorem LABEL:t:svm(LABEL:NewThe1i), Theorem 4.2(ii) and (vi), Theorem LABEL:t:tsr_suff(LABEL:t:tsr_suff_iv) and (LABEL:t:separable_and_hoelder). We emphasize the important dual characterization of subtransversality in Theorem LABEL:t:tsr_suff(LABEL:t:tsr_suff_iv) which expands and improves (in the setting adopted in the current article) (KruTha15, Theorem 4.1). The proof of this assertion is going to appear in the forthcoming article KruLukTha. In contrast to dual characterizations for transversality which are necessary and sufficient, this note underscores the fact that the known dual characterizations for subtransversality are sufficient. This raises the question whether necessary dual characterizations exist.
A projection is a selection from the projector. This exists for any closed set in Euclidean space, as can be deduced by the continuity and coercivity of the norm. Note that the projector is not, in general, single-valued, and indeed uniqueness of the projector defines a type of regularity of the set : local uniqueness characterizes prox-regularity PolRocThi00 while in finite dimensional settings global uniqueness characterizes convexity Bun34. The inverse of the projector is well defined:
Following BauLukPhaWan13.2, we use this object to define the various normal cone mappings, which in turn lead to the subdifferential of the indicator function . This brings the theory presented here to the edge of a much broader context of descent methods for solving (1). We will, however, focus exclusively on the feasibility problem for two sets.
In the above and throughout this paper, means that with .
All these three sets are clearly cones. Unlike the first two cones, the third one can be nonconvex. It is easy to verify that . Furthermore, if is closed, then
The last formula can serve as a definition of the limiting normal cone in general normed linear spaces when the original definition (5) in terms of proximal normals is not applicable. If , then . If is a convex set, then all three cones (3)–(5) coincide and reduce to the normal cone in the sense of convex analysis:
The normal space to at is defined as the orthogonal complement of and can be written as
It is in a sense a dual space object. If is a smooth manifold, then cones (3), (5) and (7) reduce to the normal space (8).
That normals to one of the sets should take into account the location of the other set was first recognized by Bauschke, Luke, Phan and Wang BauLukPhaWan13.2; BauLukPhaWan13.1 and has been used by Drusvyatskiy, Ioffe and Lewis DruIofLew14; DruIofLew15 We refer on several occasions to the preprint DruIofLew14 because some definitions and results present there and used in the current article are not included in the published version DruIofLew15., and Noll and Rondepierre NolRon15, leading to weaker “restricted” regularity conditions. The most straightforward idea is to consider only those normals to each of the sets which are directed towards the other set. Given two closed sets and and a point , one can define (see BauLukPhaWan13.2; BauLukPhaWan13.1) the following restricted analogues of the cones (3)–(5):
is metrically subregular at for if there exist and such that
If, additionally, is an isolated point of , then is called strongly metrically subregular at for .
is metrically regular at for if there exist and such that
We use the notation and to denote the supremum of all such that conditions (11) and (12), respectively, hold for some . Properties (i) and (ii) in Definition 1 are equivalent to conditions and , respectively, and the values and characterize the corresponding properties quantitatively. Some authors use for that purpose the reciprocals of the above constants (cf. DonRoc09):
which are referred to as subregularity modulus and regularity modulus, respectively. One obviously has , which means that metric regularity is in general a stronger property than metric subregularity.
Both regularity properties in Definition 1 are fundamental for variational analysis (especially the second one) and have found numerous applications in optimization and other fields. Several useful characterizations of these properties and the fundamental equivalences
have been established. We refer the readers to the monographs Mor06; DonRoc09 and surveys Aze06; Iof15; Iof00; ApeDurStr13 for a comprehensive exposition of the properties. Note that metric subregularity of at for is equivalent to the local error bound property of the real-valued function at , while metric regularity of means that the mentioned error bound property holds uniformly with respect to in a neighborhood of .
Regularity notions and convergence results
In the definitions below, we keep the terminology coming from the original publications although the use of words obviously lacks consistency.
The following definition is a compilation of some of the main definitions of regularities of sets.
(RocWet98, Definition 6.4) is Clarke regular at if .
(LewLukMal09, Definition 4.3) is super-regular at if for any , there is a such that
(PolRocThi00, Definition 1.1) is prox-regular at for if there exist such that
If is prox-regular at for all , then is said to be prox-regular at .
Convexity, of course, implies all properties in Definition 2 globally.
2 Regularity of collections of sets
The origins of the concept of regular arrangement of sets in space can be traced back to that of transversality in differential geometry which deals of course with smooth manifolds (see, for instance, GuiPol74; Hir76).
This notion has been used in LewMal08; GuiPol74; Iof15. Under this assumption, is a smooth manifold around and the following equalities hold (cf. LewMal08; GuiPol74; Iof15):
Equality (15) is only a necessary condition and is in general weaker than condition (16). When and are convex sets, it is known as the conical hull intersection property (CHIP) ChuDeuWar90 (cf. (BakDeuLi05, Definition 5.1)).
The transversality property of a collection of two smooth manifolds can be characterized quantitatively in terms of the angle between their tangent (or normal) spaces. One can use for that purpose the Friedrichs angle (cf. Deutsch Deu01).
Given two nonempty subspaces and , the Friedrichs angle is a number between 0 and whose cosine is given by
The following properties provide some insight into the geometry of the intersection (cf. (BauLukPhaWan13.2, Fact 7.10), (LewMal08, Lemmas 3.2 and 3.3)):
The angle between two smooth manifolds and around a point is defined in (LewMal08, Definition 3.1) as the Friedrichs angle between the two tangent subspaces and , or equivalently, in view of (18), the Friedrichs angle between the two normal subspaces and :
Observe that, with and , condition in (19) is equivalent to (16), and thus, to the transversality of at .
(ClaLedSteWol98, Page 99) The transversality condition holds at the intersection of two sets and if
(ZheNg08, Page 62) The collection of sets is locally linearly regular at if there exist numbers and such that
(BauLukPhaWan13.2, Definition 6.6) The -qualification condition holds at if one of the following equivalent formulations holds:
(DruIofLew14, Definition 4.4) and are inherently transversal at if there exist numbers and such that
(NolRon15, Definition 1) intersects separably at if there exist numbers and such that
If also intersects separably at , then is said to intersect separately at .
(DruIofLew15, Definition 3.1) and are intrinsically transversal at if one of the following equivalent conditions holds:
there exist numbers and such that
there exist numbers and such that
Using the Euclidean space geometry, each of the properties in Definition 4 can be reformulated equivalently in several different ways; see some reformulations in KruTha16 including the angle characterization of intrinsic transversality in (KruTha16, Proposition 19). Analytically, this means quantifying each of these properties. As observed in (LewMal08, Theorem 18), for two smooth manifolds the equalities (14) and (16) are actually equivalent. Properties (i) and (ii) are shown in Theorems 4.1 and 4.2 below to be equivalent to what we call in this article (Definition 6) transversality and subtransversality, respectively.
A more general Hölder-type setting of property (v) with exponent is considered in NolRon15. Definition 4(v) corresponds to the ‘linear’ case . Note also that this notion is not symmetric: may intersect separably, but need not intersect separably.
The known relationships between the properties in Definition 4 are as follows:
(iii) (iv) (v).
Property (vi) is in general independent of each of the properties (iii) and (iv).
When both sets are super-regular (Definition 2(v)) at the reference point, (iv) (vi).
Implications (a) and (b) follow from the description of the properties in Definition 4. The observation (c) was demonstrated in (KruTha16, Examples 23 and 24). Implication (d) was shown in (DruIofLew14, Proposition 4.5). ∎
3 Convergence results
We catalog next some of the main nonconvex convergence results making use of certain combinations of the above regularities.
For and closed with nonempty intersection and , the alternating projections algorithm converges locally linearly if one of the following collection of conditions holds.
(LewMal08, Theorem 4.3) and are smooth manifolds around and is transversal at .
(LewLukMal09, Theorem 5.16) is super-regular at and the transversality condition (21) holds at .
(BauLukPhaWan13.2, Theorem 3.17) is -regular at and the -qualification condition holds at .
(HesLuk13, Theorem 3.11) and are -subregular relative to at and is locally linearly regular at .
(DruIofLew14, Theorem 2.3) is intrinsically transversal at .
(NolRon15, Theorem 2) is -Hölder regular relative to at and intersects separably at .
For and closed with nonempty intersection and , the Douglas-Rachford algorithm converges locally linearly if one of the following collection of conditions holds.
(HesLuk13, Theorem 3.18) satisfies the transversality condition (21) at , the set is an affine subspace, and is -subregular relative to at .
(Pha15, Theorem 4.3) satisfies the transversality condition (21) at , and and are -subregular at .
Elemental set regularity
Elemental (sub)regularity defined next provides a unifying framework for the other notions of set regularity given in Definition 2.
is elementally regular of order at for with constant if there exists a neighborhood of such that, for all , is elementally subregular of order relative to at for with constant .
is uniformly elementally regular of order at for if there exists a neighborhood of such that, for all , is uniformly elementally subregular of order relative to at for .
If in (i) and (ii), then the respective qualifier, “relative to” is dropped. If , then the respective qualifier, “of order” is dropped in the description of the properties. The modulus of elemental (sub)regularity is the infimum over all for which (22) holds.
In all properties in Definition 5, need not be in and need not be in , although these are the main cases of interest for us. When , the properties are trivial for any constant , so the only case of interest is elemental (sub)regularity with constant .
The set , however, is not elementally regular at for any because by choosing (where , ), we get .
Let and suppose that there is a neighborhood of and a constant such that for each
Then, is -Hölder regular relative to at with constant and neighborhood of if and only if is elementally subregular of order relative to at for each with constant and the respective neighborhood .
The set is Clarke regular at if and only if is uniformly elementally regular at for all with . Consequently, Clarke regularity implies -regularity.
(i). The set is -Hölder regular at relative to with constant and neighborhood if and only if
holding for each and . Thanks to assumption (24), this is equivalent to being elementally subregular of order relative to at for each with constant and neighborhood .
(iii). The first part is a particular case of (ii) for . For the latter part, we suppose is -regular at and let . We can assume without loss of generality that for (otherwise one could rescale so that this holds). From the variational characterization of the Euclidean projector, if and only if
(iv). The set is Clarke regular at if and only if for any and , there is a such that
This means that is uniformly elementally subregular relative to at for all .
(v). By (LewLukMal09, Proposition 4.4), is super-regular at if and only if for any , there is a such that
(vi). By (PolRocThi00, Proposition 1.2), is prox-regular at if and only if is prox-regular at for . This means that there exist such that
(vii). Since we are in a finite dimensional setting, is nonempty closed and convex if and only if the projector is everywhere single-valued (Chebyshev (BauCom11, Theorem 3.14), Bun34, (Deu01, Theorem 12.7)) and
Assumption (24) seems to be a technical one. However, it is satisfied, for example, when the collection of sets is strongly subtransversal at with constant , where is given below in Theorem 4.1(iii), since
As a consequence, 0-Hölder regularity and elemental subregularity as specified in Proposition 4 (i) are equivalent under the additional assumption of strong subtransversality of the collection of sets. This observation falls within our interest of investigating relationships amongst various regularity notions of individual sets and collections of sets.
Regularity of collections of sets
The following definition captures two of the central notions found (under various aliases and disguises) in the literature.
is subtransversal at if there exist numbers and such that
If, additionally, is an isolated point of , then is called strongly subtransversal at . The (possibly infinite) supremum of all above is denoted with the convention that the supremum of the empty set is zero.
is transversal at if there exist numbers and such that
Definition 6(i) was introduced recently in KruTha15 and can be viewed as a local analogue of the global uniform normal property introduced in the convex setting in (BakDeuLi05, Definition 3.1(4)) as a generalization of the property (N) of convex cones by Jameson Jam72. A particular case of the Jameson property (N) for convex cones and such that and was studied by M. Krein in the 1940s. Definition 6(ii) first appeared in Kru05 (see also Kru06; Kru09) in the normed linear space setting, where the property was referred to as simply regularity (and later as strong regularity and uniform regularity). In LewLukMal09, the property is called linearly regular intersection.
If , then is trivially transversal (and consequently subregular) at with any . Thus, .
Note that, under the conditions of Example 4, does not have to be transversal at .
The next two results are a catalog of the main characterizations of subtransversality and transversality, respectively.
The following statements are equivalent to being subtransversal at .
There exist numbers and such that
There exist numbers and such that
Moreover, is the exact upper bound of all numbers such that (29) is satisfied.
There exist numbers and such that
where is the exact upper bound of all numbers such that condition (30) is satisfied.
Characterization (i). This is easily checked.
Characterization (ii). This follows from (KruTha15, Theorem 3.1).
Characterization (iii). The inequality (29) implies (30), hence, thanks to characterization (ii), subregularity of implies property (30) with the same numbers and , and the second inequality in (31) holds true.
and consequently, after passing to the limit as ,
Hence, thanks to characterization (ii), property (30) implies subtransversality of with numbers and . Because can be chosen arbitrarily close to , the first inequality in (31) holds true. This completes the proof. ∎
Thanks to characterization (ii) of Theorem 4.1, subtransversality of a collection of sets can be recognized as a well known regularity property that has been around for more than 20 years under the names of (local) linear regularity, metric regularity, linear coherence, metric inequality, and subtransversality; cf. BakDeuLi05; BauBor93; BauBor96; Iof89; Iof00; Iof15; KlaLi99; HesLuk13; LiNgPon07; NgaThe01; Pen13; ZheNg08; ZheWeiYao10; DruIofLew15; Pan15. It has been used as the key assumption when establishing linear convergence of sequences generated by cyclic projection algorithms and a qualification condition for subdifferential and normal cone calculus formulae. This property is implied by the bounded linearly regularity BauBor96. If and are closed convex sets and the collection is subtransversal at any point in , then it is boundedly linear regular; cf. (BakDeuLi05, Remark 6.1(d)). Characterization (iii) of Theorem 4.1 can be considered as a nonconvex extension of (NgYan04, Theorem 3.1).
One can also observe that condition (29) is equivalent to the function having a local error bound Aze03; FabHenKruOut10; Kru15/weak sharp minimum BurDen02; BurDen05; BurFer93 at with constant . One can think of condition (32) as a kind of uniform local error bound/relaxed weak sharp minimum property; cf. Kru06.
The following statements are equivalent to being transversal at .
There exist numbers and such that
Moreover, is the exact upper bound of all numbers such that (32) is satisfied.
There exist numbers and such that
where is the exact upper bound of all numbers such that condition (33) is satisfied.
There exists a number such that for all and with . Moreover, is the exact upper bound of all such numbers .
.
There is a number such that for all and with . Moreover, the exact lower bound of all such numbers , denoted , satisfies .